Image processing device, information storage device, and image processing method

ABSTRACT

An image sequence acquisition section acquires an image sequence including a plurality of images. A processing section performs an image summarization process that acquires a summary image sequence based on first and second deletion determination processes that delete some of the images included in the acquired image sequence. The processing section sets an attention image sequence including one at least one attention image included in the plurality of images, selects a first reference image from the attention image sequence, selects a first determination target image from the plurality of images, and performs the first deletion determination process that determines whether the first determination target image can be deleted based on first deformation information that represents deformation between the first reference image and the first determination target image. The processing section sets a partial image sequence from the image sequence, a plurality of images that have been determined to be allowed to remain by the first deletion determination process being consecutively arranged in the partial image sequence. The processing section selects a second reference image and a second determination target image from the partial image sequence, and performs the second deletion determination process that determines whether the second determination target image can be deleted based on second deformation information that represents deformation between the second reference image and the second determination target image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a Continuation application of U.S. Ser. No.14/480,570, filed on Sep. 8, 2014, which is a continuation ofInternational Patent Application No. PCT/JP2013/056273, having aninternational filing date of Mar. 7, 2013, which designated the UnitedStates, the entirety of both of which are incorporated herein byreference. Japanese Patent Application No. 2012-051559 filed on Mar. 8,2012 and Japanese Patent Application No. 2012-113618 filed on May 17,2012 are also incorporated herein by reference in their entirety.

BACKGROUND

The present invention relates to an image processing device, aninformation storage device, an image processing method, and the like.

When still images are continuously captured in time series at given timeintervals, or when a spatial object is covered by a number of images, orwhen a movie is captured, and each image that forms the movie is used asa still image, for example, a very large number of temporally orspatially continuous images (hereinafter may be referred to as “imagesequence”) are acquired. In such a case, it is likely that the imagesthat are closely situated in the image sequence (i.e., images that areclose to each other temporally or spatially) are similar images, and itis not likely that it is necessary to check all of a large number ofimages in order to determine the captured information. Since the numberof images may typically be tens of thousands or more, it takes time forthe user to check all of the images.

Therefore, it has been desired to summarize the original image sequenceusing an image sequence that includes a smaller number of images bydeleting images from the original image sequence. This process ishereinafter referred to as “image summarization process”. For example,JP-A-2009-5020 discloses an image summarization method that extracts ascene change boundary image included in the image sequence, or an imagethat represents the image sequence, and allows images from which theinformation represented by the image sequence can be easily determinedto remain.

For example, when applying the image summarization technique in themedical field, it is necessary to prevent a situation in which an areathat cannot be observed occurs due to deletion of an image in order toprevent a situation in which a disease is missed. In particular, it isnecessary to ensure that an important area such as a lesion area or anabnormal area can be reliably observed.

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing device comprising:

a first image summarization section that performs a first imagesummarization process based on a similarity between a plurality ofimages to acquire a first summary image sequence;

a second image summarization section that performs a second imagesummarization process based on a target object/scene recognition processon each image among the plurality of images to acquire a second summaryimage sequence; and an integration processing section that performs anintegration process on the first summary image sequence and the secondsummary image sequence, or performs an integration process on the firstimage summarization process and the second image summarization processto acquire an output summary image sequence.

According to another aspect of the invention, there is provided an imageprocessing device comprising:

an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and

a processing section that performs an image summarization process thatacquires a summary image sequence based on a first deletiondetermination process and a second deletion determination process thatdelete some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section, the processingsection setting an attention image sequence that includes one

attention image or a plurality of attention images included in theplurality of images, selecting a first reference image from theattention image sequence, selecting a first determination target imagefrom the plurality of images, and performing the first deletiondetermination process that determines whether or not the firstdetermination target image can be deleted based on first deformationinformation that represents deformation between the first referenceimage and the first determination target image,

the processing section setting a partial image sequence from the imagesequence, a plurality of images that have been determined to be allowedto remain by the first deletion determination process beingconsecutively arranged in the partial image sequence, and

the processing section selecting a second reference image and a seconddetermination target image from the partial image sequence, andperforming the second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as:

a first image summarization section that performs a first imagesummarization process based on a similarity between a plurality ofimages to acquire a first summary image sequence;

a second image summarization section that performs a second imagesummarization process based on a target object/scene recognition processon each image among the plurality of images to acquire a second summaryimage sequence; and

an integration processing section that performs an integration processon the first summary image sequence and the second summary imagesequence, or performs an integration process on the first imagesummarization process and the second image summarization process toacquire an output summary image sequence.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as:

an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and

a processing section that performs an image summarization process thatacquires a summary image sequence based on a first deletiondetermination process and a second deletion determination process thatdelete some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section,

the processing section setting an attention image sequence that includesone attention image or a plurality of attention images included in theplurality of images, selecting a first reference image from theattention image sequence, selecting a first determination target imagefrom the plurality of images, and performing the first deletiondetermination process that determines whether or not the firstdetermination target image can be deleted based on first deformationinformation that represents deformation between the first referenceimage and the first determination target image,

the processing section setting a partial image sequence from the imagesequence, a plurality of images that have been determined to be allowedto remain by the first deletion determination process beingconsecutively arranged in the partial image sequence, and

the processing section selecting a second reference image and a seconddetermination target image from the partial image sequence, andperforming the second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage.

According to another aspect of the invention, there is provided an imageprocessing method that performs a first image summarization processbased on a similarity between a plurality of images to acquire a firstsummary image sequence, and performs a second image summarizationprocess based on a target object/scene recognition process on each imageamong the plurality of images to acquire a second summary imagesequence, the image processing method comprising:

performing an integration process on the first summary image sequenceand the second summary image sequence, or performing an integrationprocess on the first image summarization process and the second imagesummarization process to acquire an output summary image sequence.

According to another aspect of the invention, there is provided an imageprocessing method comprising:

acquiring an image sequence that includes a plurality of images;

setting an attention image sequence that includes one attention image ora plurality of attention images included in the plurality of images;

selecting a first reference image from the attention image sequence, andselecting a first determination target image from the plurality ofimages;

performing a first deletion determination process that determineswhether or not the first determination target image can be deleted basedon first deformation information that represents deformation between thefirst reference image and the first determination target image;

setting a partial image sequence from the image sequence, a plurality ofimages that have been determined to be allowed to remain by the firstdeletion determination process being consecutively arranged in thepartial image sequence;

selecting a second reference image and a second determination targetimage from the partial image sequence;

performing a second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage; and

performing an image summarization process that deletes some of theplurality of images included in the image sequence based on the firstdeletion determination process and the second deletion determinationprocess to acquire a summary image sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configuration example of an imagesummarization device according to one embodiment of the invention.

FIG. 2 is a view illustrating a coverage ratio calculation method.

FIGS. 3A and 3B are views illustrating a specific example of a firstimage summarization process.

FIG. 4 is a flowchart illustrating a first image summarization process.

FIG. 5 is a flowchart illustrating a second image summarization process.

FIG. 6 is a view illustrating a second image summarization process.

FIG. 7 is a flowchart illustrating an integration process according to afirst embodiment.

FIGS. 8A and 8B are views illustrating a second summary image sequenceupdate process.

FIGS. 9A and 9B are views illustrating a second summary image sequenceupdate determination process.

FIG. 10 is a flowchart illustrating an integration process according toa second embodiment.

FIG. 11 illustrates another system configuration example of an imagesummarization device according to one embodiment of the invention.

FIGS. 12A to 12C are views illustrating a method according to a thirdembodiment.

FIG. 13 is a flowchart illustrating an integration process according tothe third embodiment.

FIGS. 14A to 14E are views illustrating a method according to a fourthembodiment.

FIG. 15 is a flowchart illustrating an integration process according tothe fourth embodiment.

FIGS. 16A to 16D are views illustrating an image summarization processaccording to one embodiment of the invention.

FIG. 17 illustrates a configuration example of an image processingdevice according to a fifth embodiment.

FIG. 18 is a flowchart illustrating an image summarization processaccording to the fifth embodiment.

FIGS. 19A to 19D are views illustrating a reference image/determinationtarget image selection method.

FIG. 20 illustrates a configuration example of a first deletiondetermination section.

FIG. 21 illustrates a configuration example of a second deletiondetermination section.

FIG. 22 illustrates another configuration example of a first deletiondetermination section.

FIGS. 23A to 23E are views illustrating a erosion process that utilizesa structural element and is performed on a non-coverage area.

FIGS. 24A and 24B are views illustrating a erosion process that utilizesa structural element and is performed on a determination target image.

FIGS. 25A and 25B are views illustrating an inclusion determinationprocess on a reference image and a coverage-requiring area.

FIGS. 26A and 26B are views illustrating another process that utilizes astructural element.

FIG. 27 illustrates a configuration example of a second reference imageselection section.

FIGS. 28A to 28G are views illustrating a backward reference imageupdate method.

FIG. 29 is a flowchart illustrating an image summarization processaccording to a seventh embodiment.

FIG. 30 illustrates a basic system configuration example of an imageprocessing device.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to one embodiment of the invention, there is provided an imageprocessing device comprising:

a first image summarization section that performs a first imagesummarization process based on a similarity between a plurality ofimages to acquire a first summary image sequence;

a second image summarization section that performs a second imagesummarization process based on a target object/scene recognition processon each image among the plurality of images to acquire a second summaryimage sequence; and

an integration processing section that performs an integration processon the first summary image sequence and the second summary imagesequence, or performs an integration process on the first imagesummarization process and the second image summarization process toacquire an output summary image sequence.

According to one embodiment of the invention, the integration process isperformed on the first image summarization process based on thesimilarity and the second image summarization process based on thetarget object/scene recognition process to acquire the output summaryimage sequence. This makes it possible to implement an imagesummarization process that achieves the advantages obtained when usingthe similarity and the advantages obtained when using the targetobject/scene recognition process. Therefore, it is possible to implementefficient image summarization, and improve convenience to the user, forexample.

According to another embodiment of the invention, there is provided animage processing device comprising:

an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and

a processing section that performs an image summarization process thatacquires a summary image sequence based on a first deletiondetermination process and a second deletion determination process thatdelete some of the plurality of images included in the image sequenceacquired by the image sequence acquisition section,

the processing section setting an attention image sequence that includesone attention image or a plurality of attention images included in theplurality of images, selecting a first reference image from theattention image sequence, selecting a first determination target imagefrom the plurality of images, and performing the first deletiondetermination process that determines whether or not the firstdetermination target image can be deleted based on first deformationinformation that represents deformation between the first referenceimage and the first determination target image,

the processing section setting a partial image sequence from the imagesequence, a plurality of images that have been determined to be allowedto remain by the first deletion determination process beingconsecutively arranged in the partial image sequence, and

the processing section selecting a second reference image and a seconddetermination target image from the partial image sequence, andperforming the second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage.

According to this embodiment of the invention, the attention imagesequence is set, the first deletion determination process is performedbased on the attention image sequence, and the second deletiondetermination process is performed based on the results of the firstdeletion determination process. Since the first deletion determinationprocess and the second deletion determination process are performedbased on the deformation information about images, it is possible toimplement an image summarization process that takes account of both theattention image and the deformation information. Since the process inthe subsequent stage is performed using the results of the process inthe preceding stage, the image summarization process can be effectivelyperformed as compared with the case where each process is performedindependently.

According to another embodiment of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to function as eachsection described above.

According to another embodiment of the invention, there is provided animage processing method that performs a first image summarizationprocess based on a similarity between a plurality of images to acquire afirst summary image sequence, and performs a second image summarizationprocess based on a target object/scene recognition process on each imageamong the plurality of images to acquire a second summary imagesequence, the image processing method comprising:

performing an integration process on the first summary image sequenceand the second summary image sequence, or performing an integrationprocess on the first image summarization process and the second imagesummarization process to acquire an output summary image sequence.

According to another embodiment of the invention, there is provided animage processing method comprising:

acquiring an image sequence that includes a plurality of images;

setting an attention image sequence that includes one attention image ora plurality of attention images included in the plurality of images;

selecting a first reference image from the attention image sequence, andselecting a first determination target image from the plurality ofimages;

performing a first deletion determination process that determineswhether or not the first determination target image can be deleted basedon first deformation information that represents deformation between thefirst reference image and the first determination target image;

setting a partial image sequence from the image sequence, a plurality ofimages that have been determined to be allowed to remain by the firstdeletion determination process being consecutively arranged in thepartial image sequence;

selecting a second reference image and a second determination targetimage from the partial image sequence;

performing a second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage; and

performing an image summarization process that deletes some of theplurality of images included in the image sequence based on the firstdeletion determination process and the second deletion determinationprocess to acquire a summary image sequence.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that all of the elementsdescribed in connection with the following exemplary embodiments shouldnot necessarily be taken as essential elements of the invention.

1. Method

A method used in connection with several exemplary embodiments of theinvention is described below. It is desirable to perform the imagesummarization process when an image sequence that includes a largenumber of temporally or spatially continuous images has been acquired,and the user performs a process (e.g., medical practice (e.g.,diagnosis) when the image sequence is an endoscopic image sequence)using the image sequence. This is because the number of images includedin the image sequence is very large, and it takes time for the user tocheck all of the images included in the image sequence to make adetermination. Moreover, it is likely that similar images are includedin the image sequence, and the amount of information that can beacquired is limited even if such similar images are thoroughly checked.

Specific examples of such an image sequence include an image sequencecaptured using a capsule endoscope. The capsule endoscope is acapsule-shaped endoscope that includes a small camera, and captures animage at given time intervals (e.g., twice a second). The capsuleendoscope remains inside a body for several hours (tens or more hours insome cases) until it is discharged from the body, and several tens ofthousands of captured images are acquired during a single examination.When the capsule endoscope moves inside a living body, the capsuleendoscope may stop or move backward due to the motion of the livingbody, for example. Therefore, a large number of captured images mayinclude a number of images that capture a similar object, and are notuseful for finding a lesion or the like.

A known image summarization process may extract a scene change boundaryimage or an image that represents the image sequence. However, such aknown image summarization process does not take account of therelationship between the object captured within the deletion targetimage and the object captured within the image that is allowed to remainwhen deleting an image. Therefore, the object that is captured within animage included in the image sequence that is not subjected to the imagesummarization process may not be captured within each image included inthe image sequence obtained by the image summarization process.

This is particularly undesirable when applying the image summarizationprocess in the medical field. This is because it is necessary to preventa situation in which the attention area (e.g., lesion) is missed as muchas possible. In order to prevent a situation in which the attention areais missed, it is desirable to capture a wide range inside a living body,and prevent a situation in which an object range that cannot be observedoccurs due to deletion of a given image during the image summarizationprocess.

Several aspects of the invention propose implementing an imagesummarization process from the viewpoint of preventing a situation inwhich an area that cannot be observed occurs due to deletion of animage. Specifically, the image summarization process is performed basedon the similarity between a plurality of images included in an imagesequence that is subjected to the image summarization process. It ispossible to implement the image summarization process based on therelationship between a plurality of images by utilizing the similaritybetween the plurality of images.

The similarity may be calculated in various ways. For example, areference image (i.e., an image that is allowed to remain (an image thatmay be allowed to remain depending on the reference image settingmethod)) and a determination target image (i.e., a deletiondetermination target image) may be selected from an image sequence, andthe image summarization process may be performed based on deformationinformation about the reference image and the determination targetimage. Specifically, the reference image is deformed to calculate acoverage area within the determination target image (see FIG. 2). Theobject captured within the reference image corresponds to the objectcaptured within the coverage area within the determination target image.Specifically, an area (hereinafter referred to as “non-coverage area”)within the determination target image other than the coverage areacannot be covered by the reference image when the determination targetimage is deleted.

Therefore, the degree by which an object range that cannot be observedoccurs is controlled by calculating the ratio of the coverage area withrespect to the determination target image as the coverage ratio, anddetermining whether or not to delete the determination target imagebased on the calculated coverage ratio, for example. For example, thedetermination target image is deleted when the coverage ratio is equalto or larger than a threshold value, and is not deleted when thecoverage ratio is less than the threshold value. In this case, thedegree by which an area that cannot be observed occurs can be controlledcorresponding to the threshold value.

As another example of the image summarization process that utilizes thedeformation information, whether or not the determination target imagecan be deleted may be determined based on the results of a erosionprocess on the non-coverage area utilizing a structural element(corresponding to an attention area) (see FIGS. 23A to 23E). The detailsof the erosion process are described later. In this case, at least partof an area captured within the determination target image having a sizeequal to or larger than the size of the structural element isnecessarily captured within the reference image even if thedetermination target image is deleted. Therefore, when the entireattention area is captured within the determination target image, atleast part of the attention area can be observed within the referenceimage irrespective of the position of the attention area within thedetermination target image, and a situation in which the attention areais missed can be prevented.

Note that the similarity need not necessarily be calculated using thedeformation information. The similarity between images may be calculatedusing another method.

However, since the image summarization process that utilizes thesimilarity is performed based on the relationship between the images,the object, the scene, or the like captured within the processing targetimage may not be taken into consideration.

Therefore, when it is desired to capture a specific imaging target(e.g., the observation target of the doctor when using a capsuleendoscope (a lesion or the like in a narrow sense)) within an image, itis useful to perform an image summarization process from the viewpointof whether or not the imaging target is captured within an image inaddition to the image summarization process based on the similarity.

Accordingly, several aspects of the invention propose a method thatperforms a first image summarization process based on the similarity,and a second image summarization process based on a target object/scenerecognition process, and performs an integration process that acquire anoutput summary image sequence through integration. This makes itpossible to implement an image summarization process that can achievethe advantages of the first image summarization process and the secondimage summarization process. Specifically, it is possible to implementan image summarization process that allows the user to efficientlyobserve the observation target object/scene while preventing a situationin which an object range that cannot be observed occurs.

Specific examples of the above method are described below in connectionwith first to fourth embodiments. The first embodiment illustrates anexample in which a first summary image sequence is acquired by a firstimage summarization process, a second summary image sequence is acquiredby a second image summarization process, and the first summary imagesequence and the second summary image sequence are integrated by anintegration process. The second embodiment illustrates an example inwhich the first summary image sequence and the second summary imagesequence are acquired, the second summary image sequence is updated(i.e., the number of summary images included in the second summary imagesequence is reduced (in a narrow sense)) based on the first summaryimage sequence, and the first summary image sequence and the updatedsecond summary image sequence are integrated.

The third embodiment and the fourth embodiment illustrate an example inwhich the first image summarization process and the second imagesummarization process are integrated instead of integrating the firstsummary image sequence and the second summary image sequence. In thethird embodiment, the first image summarization process is performedbased on the results of the second image summarization process (secondsummary image sequence in a narrow sense). Specifically, the summaryimage included in the first summary image sequence is determined usingthe results of the second image summarization process in addition to thesimilarity instead of determining the summary image included in thefirst summary image sequence based on the similarity (see the firstembodiment and the second embodiment).

The fourth embodiment illustrates an example in which a feedback processis performed by combining the method according to the third embodimentand the second summary image sequence update process according to thesecond embodiment. Specifically, the first image summarization processis performed based on the results of the second image summarizationprocess to acquire the first summary image sequence. The second summaryimage sequence update process is performed based on the acquired firstsummary image sequence. The first image summarization process is thenperformed based on the updated second summary image sequence to acquirethe first summary image sequence, and the acquired first summary imagesequence is set to be the output summary image sequence. When the updateprocess could not be performed, the first summary image sequence thathas been acquired is set to be the output summary image sequence (seethe third embodiment).

Note that an image sequence obtained by the first image summarizationprocess based on the similarity is referred to as “first summary imagesequence”, and each image included in the first summary image sequenceis referred to as “similarity summary image”. An image sequence obtainedby the second image summarization process based on the targetobject/scene recognition process is referred to as “second summary imagesequence”, and each image included in the second summary image sequenceis referred to as “object summary image”. An image sequence that isfinally output based on the process including the integration processand the like is referred to as “output summary image sequence”, and eachimage included in the output summary image sequence is referred to as“output summary image”.

When the process that utilizes the deformation information as thesimilarity is performed, and the attention area detection process isused as the recognition process, a method other than the methodsaccording to the first to fourth embodiments may also be employed. Thedeletion determination process that utilizes only the coverage ratiodoes not take account of whether or not the attention image (i.e., animage in which the attention area is captured) can be deleted in thesame manner as the process that utilizes only the similarity. Forexample, when the coverage ratio of the attention image by another image(e.g., another attention image or an image other than the attentionimage) is high, it is determined that the attention image can bedeleted. Therefore, all of the attention images may be deleted from theimage sequence, and it may be impossible to observe the attention areafrom the summary image sequence obtained by the image summarizationprocess.

When using the deletion determination process that utilizes thestructural element, an image in which the entire attention area iscaptured may be deleted, and an image in which only part of theattention area is captured may be allowed to remain. Such a situation isundesirable from the viewpoint of observation of the attention area.

It may be possible to perform an effective image summarization processcan be performed using the process that utilizes the deformationinformation (e.g., the process that utilizes the coverage ratio and/orthe structural element). However, when it is desired to mainly observe aspecific attention area (e.g., a lesion when using a capsule endoscope),it is useful to perform the process from the viewpoint of whether or notthe attention area is captured. Specifically, it is possible to dealwith the problem that may occur when performing the process thatutilizes the deformation information by allowing the attention image to(necessarily in a narrow sense) remain in the summary image sequence.

Therefore, several aspects of the invention propose a method that setsone image or a plurality of images included in the acquired imagesequence in which the attention area is captured to be an attentionimage sequence, and performs the image deletion determination processthat utilizes the deformation information based on the attention imagesequence to acquire the summary image sequence. However, when theattention image sequence is calculated as illustrated to FIG. 16A, andthe image sequence is independently calculated by performing the imagesummarization process based on the deformation information, the imagesmay be closely situated (see I1 in FIG. 16B) if the sum-set is merelycalculated. In this case, an image that is sufficiently covered byanother image may be allowed to remain in the summary image sequence,and the effect of reducing the number of images through the imagesummarization process may decrease. Therefore, the effect of reducingthe number of images through the image summarization process is improvedby performing a first deletion determination process based on theattention image sequence, and performing a second deletion determinationprocess based on the results of the first deletion determination process(two-step process). The first deletion determination process and thesecond deletion determination process utilize the deformationinformation, and the details thereof are described later.

An image processing device according to one embodiment of the inventionmay include a processing section 100 and an image sequence acquisitionsection 30 (see FIG. 30). The image sequence acquisition section 30acquires an image sequence that includes a plurality of images. Theprocessing section 100 sets an attention image sequence that includesone attention image or a plurality of attention images included in theplurality of images, selects a first reference image from the attentionimage sequence, selects a first determination target image from theplurality of images, and performs a first deletion determination processthat determines whether or not the first determination target image canbe deleted based on first deformation information that representsdeformation between the first reference image and the firstdetermination target image. The processing section 100 sets a partialimage sequence from the image sequence, a plurality of images that havebeen determined to be allowed to remain by the first deletiondetermination process being consecutively arranged in the partial imagesequence. The processing section 100 selects a second reference imageand a second determination target image from the partial image sequence,and performs a second deletion determination process that determineswhether or not the second determination target image can be deletedbased on second deformation information that represents deformationbetween the second reference image and the second determination targetimage.

A fifth embodiment illustrates a basic method. In the fifth embodiment,the first deletion determination process and the second deletiondetermination process are performed based on the coverage ratio. Notethat the first deletion determination process and the second deletiondetermination process may be implemented in various ways (e.g., thestructural element may be used). Such a modification will be describedin connection with a sixth embodiment. The reference image (secondreference image) and the determination target image (seconddetermination target image) used for the second deletion determinationprocess may also be selected in various ways. Such a modification willbe described in connection with a seventh embodiment.

2. First Embodiment

The method according to the first embodiment is described below. Asystem configuration example of an image summarization device will bedescribed first, and specific examples of the first image summarizationprocess and the second image summarization process will then bedescribed. The integration process will be described thereafter.

2.1 System Configuration Example

FIG. 1 illustrates a configuration example of the image summarizationdevice according to the first embodiment. As illustrated in FIG. 1, theimage summarization device includes an image sequence acquisitionsection 30, a first image summarization section 100, a second imagesummarization section 200, an integration processing section 300, and anoutput section 40. Note that the configuration of the imagesummarization device is not limited to the configuration illustrated inFIG. 1. Various modifications may be made, such as omitting some (e.g.,output section 40) of the elements illustrated in FIG. 1, or addingother elements.

The image sequence acquisition section 30 acquires image sequence datathat is subjected to the image summarization process. The image sequenceacquisition section 30 acquires a plurality of temporally or spatiallycontinuous images as the image sequence data. The image sequenceacquisition section 30 acquires the image sequence data from an imageinput device 10, an image database 20, or the like. The image inputdevice 10 may be an imaging device that captures an image (e.g., digitalcamera or capsule endoscope). The image database 20 is a database thatstores a large number of images. The image database 20 stores image dataacquired by an imaging device or the like. Note that the image database20 may be provided at position remote from the image summarizationdevice. For example, the image database 20 may be implemented by aserver or the like that is connected to the image summarization devicethrough a network. The image input device 10 and the image database 20are normally provided separately from the image summarization device.Note that the image input device 10 and the image database 20 may beincluded in the image summarization device.

The first image summarization section 100 performs the first imagesummarization process based on the similarity. The first imagesummarization section 100 may include a similarity calculation section110, a summarization section 120, and a first summary image sequencegeneration section 130. Note that the configuration of the first imagesummarization section 100 is not limited to the configurationillustrated in FIG. 1. Various modifications may be made, such asomitting some of the elements illustrated in FIG. 1, or adding otherelements.

The similarity calculation section 110 calculates the similarity betweenimages among the images included in the image sequence acquired by theimage sequence acquisition section 30. The summarization section 120performs a summarization process (i.e., a process that determines asimilarity summary image that is allowed to remain in the first summaryimage sequence, and a deletion target image) based on the calculatedsimilarity. The first summary image sequence generation section 130generates the first summary image sequence that is output from the firstimage summarization section 100 based on the summarization processperformed by the summarization section 120. Note that the details of thefirst image summarization process are described later.

The second image summarization section 200 performs the second imagesummarization process based on the target object/scene recognitionprocess. The second image summarization section 200 may include arecognition processing section 210, a summarization section 220, and asecond summary image sequence generation section 230. Note that theconfiguration of the second image summarization section 200 is notlimited to the configuration illustrated in FIG. 1. Variousmodifications may be made, such as omitting some of the elementsillustrated in FIG. 1, or adding other elements.

The recognition processing section 210 performs a recognition processthat determines whether or not the image included in the image sequenceacquired by the image sequence acquisition section 30 includes thetarget object, or determines whether or not the image included in theimage sequence acquired by the image sequence acquisition section 30captures the target scene. The recognition process may be implemented invarious ways. For example, a template that represents the target objector the target scene may be stored, and a matching process that utilizesthe template may be performed. The summarization section 220 performs asummarization process (i.e., a process that determines an object summaryimage that is allowed to remain in the second summary image sequence,and a deletion target image) based on the recognition results of therecognition processing section 210. Specifically, the summarizationsection 220 performs a segmentation process taking account of an area inwhich an identical target object or an identical scene is continuouslycaptured, and selects at least one image from the generated segment(consecutive image sequence) as the object summary image. The secondsummary image sequence generation section 230 generates the secondsummary image sequence that is output from the second imagesummarization section 200 based on the summarization process performedby the summarization section 220. Note that the details of the secondimage summarization process are described later.

The integration processing section 300 performs the integration processbased on the process performed by the first image summarization section100 and the process performed by the second image summarization section200. In the first embodiment, the integration processing section 300performs the integration process on the first summary image sequence andthe second summary image sequence. The details thereof are describedlater.

The output section 40 outputs the output summary image sequence acquiredas the results of the integration process performed by the integrationprocessing section 300. The output section 40 may be a display sectionthat is implemented by a liquid crystal display, an organic EL display,or the like. In this case, the output section 40 may display the outputsummary image included in the output summary image sequence, forexample. Note that the image summarization device need not necessarilyinclude a display section or the like that serves as an interface withthe user. The output section 40 (display section) may be providedseparately from the image summarization device.

2.2 First Image Summarization Process

The first image summarization process based on the similarity isdescribed below. The similarity may be the motion vector between images,the SSD, the SAD, a correlation value (e.g., normalizedcross-correlation), or the like. Arbitrary information may be used asthe similarity as long as the information is normally calculated as thesimilarity between a plurality of images.

The image summarization process based on the similarity may beimplemented using a known method that detects a scene change byperforming a sorting process in ascending order of the similarity, andperforming a selection process up to the set number.

As illustrated in FIG. 2, the coverage ratio of the determination targetimage by the reference image (i.e., the similarity summary image that isallowed to remain in the first summary image sequence, or a candidateimage for the similarity summary image) may be used as the similarity,and whether or not the determination target image can be deleted may bedetermined based on the coverage ratio to implement the imagesummarization process. The method that utilizes the coverage ratio isdescribed in detail below.

The method that utilizes the coverage ratio deforms the reference imageusing the deformation information about the reference image and thedetermination target image, and projects the reference image onto thedetermination target image. The deformation information refers toinformation that represents a state in which the object captured withinthe reference image is deformed within the determination target image.The deformation information may be calculated from deformationestimation, the motion vector, or the like, or a non-rigid deformationparameter estimated by the method disclosed in JP-A-2011-24763 may beused as the deformation information, for example.

FIG. 2 illustrates an example in which a first reference image thatprecedes the determination target image, and a second reference imagethat follows the determination target image are set to be the referenceimage. An area within the determination target image that is indicatedby A1 is an area obtained by deforming the first reference image, and anarea within the determination target image that is indicated by A2 is anarea obtained by deforming the second reference image. In this case, anarea that corresponds to the sum-set of the area indicated by A1 and thearea indicated by A2 may be calculated as the coverage area, and theratio of the coverage area to the entire determination target image maybe used as the coverage ratio, for example.

Whether or not the determination target image can be deleted may bedetermined by comparing the coverage ratio with a threshold value thatis set in advance. Note that the threshold value may be set by thesystem, or may be determined based on an input performed by the user.Specifically, when the coverage ratio is less than the threshold value,it is determined that the determination target image cannot be deleted.When the coverage ratio is equal to or more than the threshold value, itis determined that the determination target image can be deleted. Whenthe coverage ratio is equal to or more than the threshold value, an areaof the object range captured within the determination target image thatis represented by the threshold value is captured within at least one ofthe first reference image and the second reference image. Therefore,when the first reference image and the second reference image areallowed to remain as the similarity summary image, the area capturedwithin the determination target image is sufficiently covered even whenthe determination target image is deleted.

FIGS. 3A and 3B illustrate the process that selects the first referenceimage, the second reference image, and the determination target image.Note that it has been determined that the first reference image isselected as the similarity summary image. On the other hand, the secondreference image is a candidate for the similarity summary image, and ithas not been determined that the second reference image is selected asthe similarity summary image.

In FIG. 3A, the kth image of the image sequence is selected as the firstreference image. The first to (k−1)th images have been determined to beselected as the similarity summary image, or deleted, and the kth to Nthmages are the processing target. In this case, the (k+2)th image isselected as the second reference image.

The determination target image is sequentially selected from the firstimage among the images situated between the first reference image andthe second reference image. The first reference image is deformed basedon the deformation information about the first reference image and thedetermination target image, and the second reference image is deformedbased on the deformation information about the second reference imageand the determination target image to calculate the coverage ratio.Whether or not the determination target image can be deleted isdetermined based on the calculated coverage ratio.

When it has been determined that all of the images situated between thefirst reference image and the second reference image can be deleted (seeFIG. 3A) (threshold value=70%) (i.e., the image that follows the currentsecond reference image can be selected as the second reference image),the next second reference image is selected as illustrated in FIG. 3B.Specifically, the (k+1)th image is selected as the next second referenceimage.

Whether or not the images situated between the first reference image andthe second reference image can be deleted is then determined. When ithas been determined that the determination target image cannot bedeleted (see FIG. 3B) (i.e., all of the images situated between thefirst reference image and the current second reference image are notcovered by the first reference image and the current second referenceimage) (i.e., the determination target image that cannot be deleted isnot covered by the first reference image and the current secondreference image), it is considered that the update of the secondreference image (increment in the selection position) was inappropriate(i.e., the interval between the first reference image and the secondreference image was increased to a large extent).

Therefore, the image that immediately precedes the current secondreference image (corresponding to the second reference image in FIG. 3A)is allowed to remain as the similarity summary image. Specifically, theimage that immediately precedes the current second reference image isselected as the next first reference image, and the second referenceimage and the determination target image are selected to continue theprocess.

FIG. 4 is a flowchart illustrating the first image summarization processwhen using the coverage ratio as the similarity. When the first imagesummarization process has started, the I_(S)th image of the imagesequence that is subjected to the image summarization process is set tobe the first similarity summary image (S101). The value I_(s) may be 1(i.e., the first image of the image sequence may be set to be thesimilarity summary image), or may be a value other than 1.

The I_(S)th image is set to be the first reference image (S102), and theI_(E)th images is set to be the second reference image (S103). Theinitial value of the value I_(E) is a value that satisfiesI_(E)=I_(S)+2.

Whether or not the value I_(E) is larger than the number of imagesincluded in the processing target image sequence is determined (S104).When it has been determined that the value I_(E) is larger than thenumber of images included in the processing target image sequence, theimage summarization process is terminated. When it has been determinedthat the value I_(E) is not larger than the number of images included inthe processing target image sequence (i.e., the second reference imagecould be appropriately set), the images situated between the firstreference image and the second reference image are sequentially set tobe the determination target image, and whether or not the determinationtarget image can be deleted is determined. Specifically, the image thatimmediately follows the first reference image is set to be the firstdetermination target image, and the coverage ratio is calculated usingthe method illustrated in FIG. 2, and compared with the threshold value(S106 and S107). When the coverage ratio is equal to or more than thethreshold value (Yes in S107) (i.e., the determination target image canbe deleted), the image that immediately follows the currentdetermination target image is set to be the determination target image(i.e., the value i is incremented in FIG. 4). When it has beendetermined that the coverage ratio is equal to or more than thethreshold value (Yes in S107), and the loop process in the steps S105 toS108 has ended (i.e., all of the images situated between the firstreference image and the second reference image can be covered by thefirst reference image and the second reference image) (see FIG. 3A), thevalue I_(E) is incremented to update the second reference image (S109),and the step S104 is performed.

When it has been determined that the coverage ratio is less than thethreshold value (No in S107) (i.e., at least one image among the imagessituated between the first reference image and the second referenceimage cannot be sufficiently covered by the first reference image andthe second reference image) (see FIG. 3B), it is necessary to allow theimage that immediately precedes the current second reference image toremain as the summary image. Therefore, the (I_(E)−1)th image is set tobe the next similarity summary image (S110). The value I_(S) is set toI_(E)−1 (S111), the image set to be the similarity summary image is setto be the next first reference image (S102), and the process isperformed again.

2.3 Second Image Summarization Process

The second image summarization process based on the target object/scenerecognition process is described below. The recognition process mayutilize the processing results of various image recognition/imagedetection processes (e.g., the processing results of a detection processbased on the similarity with the reference image, or the recognitionresults obtained by a pattern recognition process using machinelearning).

In the second image summarization process, the recognition process isperformed on each image of the image sequence that is subjected to theimage summarization process to determine whether or not the targetobject is captured within each image, or determine whether or not thetarget scene is captured within each image. Consecutive images among theimages in which the target object is captured, or consecutive imagesamong the images in which the target scene is captured, are set to bethe consecutive image sequence (segment). At least one image isextracted from each segment, and set to be the object summary image thatis allowed to remain in the second summary image sequence.

FIG. 5 is a flowchart illustrating the second image summarizationprocess based on the target object/scene recognition process. Althoughan example in which an image in which the recognized target object hasthe maximum area is selected as the object summary image is describedbelow, the method that selects the object summary image from theconsecutive image sequence is not limited thereto.

Note that the recognition process has been performed before the processillustrated in FIG. 5 is performed. An ID is sequentially assigned tothe images in which the target object or the like has been detected. TheID is linked to a number that represents the position of the image inthe image sequence (input image sequence) that is subjected to the imagesummarization process. In FIG. 6, the target object or the like has beendetected within the images indicated by the diagonal lines by therecognition process performed on the input image sequence, and an ID issequentially assigned from the first image among the images indicated bythe diagonal lines, for example. Note that the image number and the IDneed not necessarily start from 0. Since the position of each image (towhich the ID is assigned) in the input image sequence can be determined,information is stored that indicates that the image to which the ID=0 isassigned is the first image of the input image sequence image, and theimage to which the ID=4 is assigned is the seventh image of the inputimage sequence image. FIG. 5 illustrates the subsequent segmentationprocess and representative image (summary image) selection process.

When the second image summarization process has started, a counter valuecount is initialized (S201). The counter value count corresponds to thenumber of object summary images. When one summary image is selected fromone segment, the counter value count also corresponds to the segmentthat is set as a result of the segmentation process. A variable max thatrepresents the maximum area of the target object is initialized (S202).

After initialization, a loop process (S203 to S208) is repeated tosequentially process the images to which the ID is assigned.Specifically, the initial value is set to J=0, and the area of thetarget object within the image to which the ID=j is assigned is comparedwith the variable max. When the area of the target object is larger thanthe variable max, the variable max is overwritten with the area of thetarget object, and the image to which the ID=j is assigned is set to bethe summary image corresponding to the counter value count (S204). Sinceit is desirable to designate the summary image using the number in theinput image sequence instead of using the ID value, information isstored that indicates the image of the input image sequence thatcorresponds to the summary image corresponding to the counter valuecount based on the relationship between the image number in the inputimage sequence and the ID.

Whether or not the image to which the ID=j is assigned is situatedadjacent to the image of the input image sequence to which the ID=j+1 isassigned is determined (S205). It is determined that the image to whichthe ID=j is assigned is not situated adjacent to the image to which theID=j+1 is assigned (No in S205) when the processing target is the lastimage of the segment (e.g., ID=2 or ID=4 in FIG. 6). Therefore, theprocess on the current segment is terminated, the counter value count isincremented (S206), and the variable max is initialized (S207) (i.e., apreliminary process on the next segment).

It is determined that the image to which the ID=j is assigned issituated adjacent to the image to which the ID=j+1 is assigned (Yes inS205) when the processing target is the first image or the intermediateimage of the segment (e.g., ID=1 or ID=3 in FIG. 6). In this case, thesteps S206 and S207 are not performed. When the processing target is thefirst image of the segment (max=0), the current image is selected in thestep S204 as a provisional object summary image. When the processingtarget is the intermediate image of the segment, a given image that isincluded in the current segment and precedes the processing target isprovisionally selected as the object summary image corresponding to thecounter value count, and the area of the target object is stored as thevariable max. Therefore, the area of the target object within theprovisional object summary image is compared in the step S204 with thearea of the target object within the image to which the ID=j isassigned. When the area of the target object within the image to whichthe ID=j is assigned is larger than the area of the target object withinthe provisional object summary image, the object summary imagecorresponding to the counter value count is overwritten with the imageto which the ID=j is assigned. When the area of the target object withinthe provisional object summary image is larger than the area of thetarget object within the image to which the ID=j is assigned, the objectsummary image is maintained.

Specifically, the loop process (S203 to S208) performs the segmentationprocess, and selects the image of each segment in which the targetobject has the maximum area as the object summary image. When the aboveprocess has been performed on each image in which the target object hasbeen detected, the process is terminated.

Although an example in which the image in which the target object hasthe maximum area is selected as the object summary image has beendescribed above, the image summarization process may be similarlyperformed utilizing information about the results of the imagerecognition process or the image detection process (e.g., the positionof the target object within the image, color information, textureinformation, or recognition/detection accuracy).

2.4 Integration Process

The integration process according to the first embodiment is describedbelow. The integration process selects an image that is included in atleast one of the first summary image sequence obtained by the firstimage summarization process and the second summary image sequenceobtained by the second image summarization process as the output summaryimage of the output summary image sequence.

FIG. 7 is a flowchart illustrating the integration process according tothe first embodiment. As illustrated in FIG. 7, a loop process (S301 toS306) is performed during the integration process. The steps S302 toS305 are performed on all of the images of the input image sequence.

Specifically, whether or not the ith image of the input image sequenceis included in the first summary image sequence (i.e., whether or notthe ith image is the similarity summary image) is determined (S302).When the ith image is included in the first summary image sequence (Yesin S302), the ith image is selected as the output summary image of theoutput summary image sequence (S303). When the ith image is not includedin the first summary image sequence (No in S302), whether or not the ithimage is included in the second summary image sequence (i.e., whether ornot the ith image is the object summary image) is determined (S304).When the ith image is included in the second summary image sequence (Yesin S304), the ith image is selected as the output summary image (S303).When the ith image is not included in the second summary image sequence(No in S304), the ith image is determined to be the deletion targetimage (S305). The above process is performed on the entire input imagesequence while incrementing the value i (initial value: i=0).

According to the first embodiment, the image summarization deviceincludes the first image summarization section 100 that performs thefirst image summarization process based on the similarity between aplurality of images to acquire the first summary image sequence, thesecond image summarization section 200 that performs the second imagesummarization process based on the target object/scene recognitionprocess on each image among the plurality of images to acquire thesecond summary image sequence, and the integration processing section300 that performs the integration process on the first summary imagesequence and the second summary image sequence, or performs theintegration process on the first image summarization process and thesecond image summarization process to acquire the output summary imagesequence (see FIG. 1).

The plurality of images normally correspond to the entire image sequenceacquired by the image sequence acquisition section 30. Note that theplurality of images may be part of the image sequence acquired by theimage sequence acquisition section 30.

According to this configuration, since the summary image sequenceobtained by the image summarization process based on the determinationas to the similarity between the images, and the summary image sequenceobtained by the image summarization process based on the targetobject/scene recognition process, can be integrated to acquire theoutput summary image sequence, it is possible to implement an imagesummarization process that achieves the advantages of each imagesummarization process. A summary image sequence that includes anappropriate image in which an important target object is captured can begenerated by performing the second image summarization process thatutilizes the target object/scene recognition process. However, it isdifficult to detect all of the important target objects using the imagerecognition/detection process. It is possible to cause an importantobject that cannot be detected to be included in the first summary imagesequence acquired by the first image summarization process by preventinga situation in which an area that cannot be observed occurs through theimage summarization process based on the similarity (i.e., a differentviewpoint), and generate a complementary output summary image sequencevia the integration process.

The first image summarization section 100 may select the reference imageand the determination target image from the plurality of images,calculate the coverage ratio of the determination target image by thereference image as the similarity based on the deformation informationabout the reference image and the determination target image, andperform the first image summarization process that determines whether ornot the determination target image can be deleted based on the coverageratio.

The coverage ratio is information that represents the degree by whichthe object captured within the determination target image is capturedwithin the reference image. For example, when an image having an aspectratio of 1:1 is acquired, a 10×10 m (dimensions in the real space)square object is captured over the entire determination target image,and a 5×5 m square object that is included in the 10×10 m square objectis captured over the entire reference image. In this case, a 100 m²(area in the real space) area is captured within the determinationtarget image, and a 25 m² (area in the real space) area (that isincluded in the 100 m² area) is captured within the reference image.Therefore, the reference image covers 25% of the determination targetimage. In this case, the coverage ratio is 25%, 25 m², or 0.25, forexample. Since a planar object is rarely captured almostperpendicularly, the reference image and the determination target imagenormally differ in the shape of the object even if an identical objectis captured within the reference image and the determination targetimage. According to the first embodiment, the deformation informationcorresponding to such a deformation is acquired using the methoddisclosed in JP-A-2011-24763 or the like, and the coverage ratio iscalculated using the deformation information. Note that the coverageratio is information that represents the degree of coverage of thedetermination target image by the reference image, and is not limited toa ratio and the like.

Whether or not the determination target image can be deleted isdetermined by performing a comparison process using a given thresholdvalue, for example. It is expected that the degree by which an area thatcannot be observed occurs due to deletion of an image can be reduced byincreasing the threshold value (e.g., setting the threshold value to avalue close to 100%). On the other hand, the number of images includedin the summary image sequence can be reduced by decreasing the thresholdvalue. Since the effect of reducing the degree by which an area thatcannot be observed occurs due to deletion of an image and the effect ofreducing the number of images included in the summary image sequencehave a trade-off relationship, and can be controlled by appropriatelysetting the threshold value, it is desirable to appropriately set thethreshold value corresponding to the situation.

The above configuration makes it possible to prevent a situation inwhich an object area that cannot be observed occurs due to deletion ofan image during the first image summarization process, and control thedegree by which occurrence of such an object area is prevented.Specifically, when a value that corresponds to x % is used as thethreshold value used to determine whether or not the determinationtarget image can be deleted (deletion determination process), the methodaccording to the first embodiment ensures that x % of the objectcaptured within the determination target image is covered by thereference image even when the determination target image is deleted.Note that an area of the determination target image that is covered bythe reference image may be less than x % even when a value thatcorresponds to x % is used as the threshold value since it is difficultto accurately calculate the deformation of the object within the imageas the deformation information without an error.

When the first to Nth (N is an integer equal to or larger than 2) imageshave been input as the input image sequence, the first imagesummarization section 100 may select the pth image as the firstreference image, select the qth (q is an integer that satisfiesp+2≦q≦N−1) image as the second reference image, and select the rth (r isan integer that satisfies p+1≦r≦q−1) image as the determination targetimage. The first image summarization section 100 may calculate thecoverage ratio based on the deformation information about the firstreference image and the determination target image and the deformationinformation about the second reference image and the determinationtarget image, and determine whether or not the determination targetimage can be deleted based on the calculated coverage ratio. When it hasbeen determined that the (p+1)th to (q−1)th images can be deleted, thefirst image summarization section 100 may select the (q+1)th image asthe second reference image.

This makes it possible to set the reference images to precede or followthe determination target image (see FIGS. 3A and 3B), and implement theimage summarization process based on the coverage ratio. Since tworeference images are used, it is likely that it is determined that thedetermination target image can be deleted as compared with the casewhere one reference image is set, for example, and the number of imagesincluded in the summary image sequence (the number of images after thesummarization process) can be reduced.

The first image summarization section 100 may allow the image selectedas the first reference image to remain in the first summary imagesequence when it has been determined that at least one image among the(p+1)th to (q−1)th images cannot be deleted. The first imagesummarization section 100 may select the (q−1)th image as the firstreference image, and perform the process again.

This makes it possible to allow the first reference image to remain inthe first summary image sequence. Since the case where at least one ofthe determination target images situated between the first referenceimage and the second reference image cannot be deleted corresponds tothe case where the interval between the first reference image and thesecond reference image is increased to a large extent, it is necessaryto allow the image that precedes (immediately precedes in a narrowsense) the second reference image to remain in the first summary imagesequence. Therefore, the (q−1)th image is selected as the next firstreference image, and the process is performed again.

Note that the second reference image selection (update) process duringthe first image summarization process is not limited to the method thatsequentially shifts the second reference image to the image of the inputimage sequence that follows the current second reference image.

For example, the second reference image is selected from the secondreference image selection interval in which the starting point and theend point are set corresponding to the (p+2)th to Nth images, andwhether or not the determination target image can be deleted isdetermined based on the first reference image and the second referenceimage. In this case, the first image summarization section 100 mayselect the xth (x is an integer that satisfies x>q) image included inthe second reference image selection interval as the next secondreference image when it has been determined that the (p+1)th to (q−1)thimages can be deleted, and update the starting point of the secondreference image selection interval with the qth image. The first imagesummarization section 100 may select the yth (y is an integer thatsatisfies y<q) image included in the second reference image selectioninterval as the next second reference image when it has been determinedthat at least one image among the (p+1)th to (q−1)th images cannot bedeleted, and update the end point of the second reference imageselection interval with the qth image.

The second reference image selection interval represents a candidate forthe second reference image, and represents a candidate for thesimilarity summary image that is allowed to remain in the first summaryimage sequence (i.e., the next similarity summary image that follows thesimilarity summary image that has been found in a narrow sense).Therefore, the second reference image selection interval corresponds tothe similarity summary image search range.

This makes it possible to flexibly determine the position of the nextsecond reference image when updating the second reference image. Sincethe method that sequentially shifts the second reference image to theimage that follows the current second reference image narrows the searchrange by thoroughly checking the search range from the first image, theamount of calculations may significantly increase depending on theposition of the correct answer. In contrast, the search range can besignificantly narrowed by the unit determination (one second referenceimage selection process and one deletion determination process) byallowing a non-adjacent image to be selected as the next secondreference image. This makes it possible to reduce the amount ofcalculations, and reduce the load imposed on the system, or reduce theprocessing time. Since the backward search process is not limited to aprocess that selects the adjacent image, the range that precedes thecurrent second reference image may not have been searched, and mayinclude a correct answer depending on the deletion determination result.The forward search process is also enabled taking account of such asituation, and is not limited to a process that selects the adjacentimage as the second reference image.

When the jth (j is an integer) image corresponds to the end point of thesecond reference image selection interval, the first image summarizationsection 100 may set the value x based on the value (q+j)/2.Alternatively, when the ith (i is an integer) image corresponds to thestarting point of the second reference image selection interval, thefirst image summarization section 100 may set the value y based on thevalue (i+q)/2.

This makes it possible to use the binary search method when selectingthe next second reference image. The image that is situated between thecurrent second reference image and the end point is selected whenperforming the backward search process, and the image that is situatedbetween the current second reference image and the starting point isselected when performing the forward search process. This makes itpossible to halve the search range (corresponding to the length of thesecond reference image selection interval). It is expected that theentire search range is completely searched when log N images areselected as the second reference image. Therefore, the amount ofcalculations can be reduced to N×log N. When N is very large, the amountof calculations can be significantly reduced as compared with the methodthat sequentially shifts the second reference image to the image thatfollows the current second reference image (the amount of calculationsis N²). Note that the value (q+j)/2 and the value (i+q)/2 are notnecessarily an integer, and an image corresponding to each value may beabsent. In such a case, the maximum integer that does not exceed thevalue (q+j)/2, or an integer that is larger than the value (q+j)/2 by 1may be used, for example. Note that the second reference image need notnecessarily be updated using the binary search method, but may beupdated using various other methods. For example, when the position of acorrect answer within the search range is predicted to some extent, theamount of calculations is expected to be reduced by selecting the secondreference image so that the predicted position and its peripheralpositions can be mainly searched.

The first image summarization section 100 may calculate the coveragearea that is an area in which the determination target image is coveredby the reference image based on the deformation information about thereference image and the determination target image, and calculate theratio of the coverage area to the determination target image as thecoverage ratio.

This makes it possible to calculate the coverage ratio based on thecoverage area. The coverage area is the area illustrated in FIG. 2. Thecoverage area is an area obtained by deforming the reference image basedon the deformation information, and projecting the deformed referenceimage onto the determination target image. The object area capturedwithin the reference image corresponds to (or coincides with (when thedeformation information includes no error (ideal situation))) the objectarea captured within the calculated coverage area. Therefore, thecoverage ratio can be calculated from the ratio (i.e., area ratio) ofthe coverage area to the determination target image. Note that thecoverage area is calculated by deforming the reference image based onthe deformation information, and the calculated coverage area need notnecessarily be projected onto the determination target image. Thecoverage area need not necessarily be calculated based on the entiretyof the reference image. The coverage area may be calculated by deformingpart of the reference image based on the deformation information.

The second image summarization section 200 may set consecutive imagesamong the plurality of images that include an identical target object,or consecutive images among the plurality of images that have beenrecognized to be an identical scene, to be the consecutive imagesequence from which the summary image is extracted, based on therecognition process, and perform the second image summarization processthat extracts at least one image from the consecutive image sequence asthe summary image.

Note that at least one image extracted from the consecutive imagesequence as the summary image refers to the object summary image that isallowed to remain in the second summary image sequence.

This makes it possible to implement the second image summarizationprocess using the method that sets the consecutive image sequence(segment) described above with reference to FIGS. 5 and 6. Since theconsecutive image sequence includes consecutive images that include anidentical target object, or consecutive images that have been recognizedto be an identical scene, it is possible to reduce the image redundancywhile preventing a situation in which the target object or the scene isnot included in the summary image sequence, by extracting images fromthe consecutive image sequence in a number smaller than the number ofimages included in the consecutive image sequence as the object summaryimage. Note that the redundancy can be further reduced by reducing thenumber of images extracted from the consecutive image sequence as thesummary image. For example, only one image may be extracted from theconsecutive image sequence as the summary image.

The second image summarization section 100 may select the summary image(object summary image) extracted from the consecutive image sequencebased on the area of the target object.

According to this configuration, since an image in which the targetobject is captured to occupy a large area can be extracted as thesummary image, observation by the user can be facilitated, for example.The user may not be able to easily observe the target object even if thetarget object has a large area, when the target object is dark due to asmall brightness value, when the target object has an extreme shape andis not suitable for observation, or when the target object is situatedin the peripheral area of the image, and affected by distortion to alarge extent, for example. Therefore, information about the results ofthe image recognition process or the image detection process (e.g., theposition of the target object within the image, color information,texture information, or recognition/detection accuracy) may be used inaddition to the area of the target object. In this case, since the imagesummarization process can be performed taking account of the imagefeatures of the target object, an image in which the detected targetobject can be easily observed can be selected as the summary image, andthe target object can be more easily determined.

The integration processing section 300 may perform the integrationprocess on the first summary image sequence and the second summary imagesequence by selecting an image that is included in at least one of thefirst summary image sequence and the second summary image sequence asthe summary image (output summary image) of the output summary imagesequence.

This makes it possible to implement the integration process illustratedin FIG. 7. Since the integration process according to the firstembodiment can be implemented using a simple method (see FIG. 7(flowchart)), the processing load can be reduced.

The first image summarization section 100 may detect a scene changebased on the similarity between the plurality of images, and perform thefirst image summarization process based on the scene change.

According to this configuration, since consecutive similar images aredeleted by performing the image summarization process based on a scenechange, redundant similar scenes can be deleted, and the summary imagesequence can be generated while efficiently reducing the number ofimages. Since the above process utilizes the similarity between aplurality of images, it suffices to detect that the a first scenecaptured within a first image differs from a second scene capturedwithin a second image, and it is unnecessary to determine a specificscene that corresponds to the first scene or the second scene. On theother hand, the scene recognition process during the second imagesummarization process must determine whether or not the scene capturedwithin the processing target image coincides with the detection targetscene, and store the feature quantity and the like of the detectiontarget scene, for example.

The plurality of images may be a capsule endoscopic image. The secondimage summarization section 200 may perform the recognition process onan in vivo attention area captured within the capsule endoscopic imageas the target object.

The term “attention area” used herein refers to an area for which theuser's observation priority is relatively higher than that of otherareas. For example, when the user is a doctor, and desires to performtreatment, the attention area refers to an area in which a mucousmembrane area or a lesion area is captured. If the doctor desires toobserve bubbles or feces, the attention area refers to an area in whicha bubble area or a feces area is captured. Specifically, the attentionarea for the user differs depending on the objective of observation, butis necessarily an area for which the user's observation priority isrelatively higher than that of other areas.

This makes it possible to apply the image summarization processaccording to the first embodiment to an image captured using a capsuleendoscope. In the medical field, it is necessary to prevent a situationin which a lesion or the like is missed as much as possible. When theattention area is used as the target object during the second imagesummarization process, it is possible to efficiently allow an image inwhich the attention area has been detected to remain in the summaryimage sequence. However, the attention area is not necessarily detectedsuccessfully. Therefore, the target object may not be detected by therecognition process from an image in which the attention area iscaptured, and the image may be deleted without being determined to be acandidate for the object summary image. Accordingly, it is advantageousto use the image summarization process that utilizes the similarity(coverage ratio in a narrow sense) in combination with the aboveprocess. In particular, since it is difficult for the doctor toexternally operate a capsule endoscope that is situated inside the body,and efficiently change the imaging target, a large number of similarimages may be acquired. The process is normally performed after a largenumber of images have been stored instead of sequentially checking thecaptured image in real time since the operation is difficult. Therefore,it is advantageous to perform the image summarization process accordingto the first embodiment on an image sequence acquired by a capsuleendoscope.

The second image summarization section 200 may perform the recognitionprocess on the in vivo attention area based on a special light imageacquired by applying light within a specific wavelength band.

According to this configuration, since observation can be performedusing the special light, the target object detection accuracy during thesecond image summarization process can be improved, and it is possibleto prevent a situation in which an image in which an important object(attention area) is captured is deleted by the image summarizationprocess. Note that it is desirable to use the first image summarizationprocess that utilizes the similarity in combination with the aboveprocess even if the target object detection accuracy can be improved. Inthis case, it is possible to improve the effect of preventing asituation in which the attention area is missed.

The specific wavelength band may be a band that is narrower than thewavelength band of white light. Specifically, the special light imagemay be an in vivo image, and the specific wavelength band may be thewavelength band of light absorbed by hemoglobin in blood. Morespecifically, the specific wavelength band may be a wavelength band of390 to 445 nm or 530 to 550 nm. This corresponds to narrow band imaging(NBI).

This makes it possible to observe the structure of a surface area of invivo tissue and a blood vessel situated in a deep area. A lesion (e.g.,epidermoid cancer) or the like that cannot be easily observed usingnormal light can be displayed in brown or the like by inputting theresulting signal to a specific channel (R, G, or B), so that a situationin which a lesion area is missed can be prevented. The wavelength bandof 390 to 445 nm or 530 to 550 nm is selected from the viewpoint ofabsorption by hemoglobin and the ability to reach a surface area or adeep area of tissue.

Note that the light having the specific wavelength band is not limitedto light corresponding to NBI, but may be light corresponding toautofluorescence imaging (AFI) or infrared imaging (IRI).

The first embodiment may also be applied to a program that causes acomputer to function as the first image summarization section 100 thatperforms the first image summarization process based on the similaritybetween a plurality of images to acquire the first summary imagesequence, the second image summarization section 200 that performs thesecond image summarization process based on the target object/scenerecognition process on each image among the plurality of images toacquire the second summary image sequence, and the integrationprocessing section 300 that performs the integration process on thefirst summary image sequence and the second summary image sequence, orperforms the integration process on the first image summarizationprocess and the second image summarization process to acquire the outputsummary image sequence.

This makes it possible to implement a program that implements the aboveimage summarization process. For example, when the image summarizationprocess is implemented by an information processing system such as a PC,the program is read and executed by a processing section (e.g., CPU orGPU) included in the PC. The program is stored in an information storagemedium. The information storage medium may be an arbitrary recordingmedium that is readable by an information processing system (e.g., PC),such as an optical disk (e.g., DVD and CD), a magnetooptical disk, ahard disk (HDD), or a memory (e.g., nonvolatile memory and RAM).

1. Second Embodiment

The second embodiment is described below. A system configuration exampleof an image summarization device according to the second embodiment isthe same as that illustrated in FIG. 1 (see the first embodiment), anddetailed description thereof is omitted. The second embodiment differsfrom the first embodiment as to the integration process performed by theintegration processing section 300. The difference from the firstembodiment is described in detail below.

The first summary image sequence is acquired by the first imagesummarization process, and the second summary image sequence is acquiredby the second image summarization process in the same manner asdescribed above in connection with the first embodiment. In the secondembodiment, the second summary image sequence is updated based on thefirst summary image sequence before integrating the first summary imagesequence and the second summary image sequence.

FIGS. 8A and 8B are views illustrating the integration process accordingto the second embodiment. In FIG. 8A, the continuous horizontal straightline indicates the input image sequence (i.e., a plurality of imagesincluded in the input image sequence). Each vertical line indicates theobject summary image selected as a result of the second imagesummarization process. Each horizontal arrow in FIG. 8A indicates thatthe target object or the like was successively detected in the imagesincluded in the range. Each arrow corresponds to the consecutive imagesequence (segment). In FIG. 8B, the continuous horizontal straight lineindicates the input image sequence, and each vertical line indicates thesimilarity summary image.

The integration process according to the second embodiment integrates(combines) a plurality of consecutive image sequences that have been setas a result of the second image summarization process into oneintegrated (combined) consecutive image sequence, and extracts at leastone summary image from the integrated consecutive image sequence toreduce the number of object summary images included in the secondsummary image sequence.

A specific example of the integration process according to the secondembodiment is described below. Two adjacent images among the similaritysummary images included in the first summary image sequence areselected. When a plurality of consecutive image sequences set by thesecond image summarization process are included between the twosimilarity summary images, whether or not the plurality of consecutiveimage sequences can be integrated is determined.

Alternatively, adjacent consecutive image sequences may be selectedinstead of using adjacent similarity summary images, and whether or notthe consecutive image sequences can be integrated may be determinedbased on whether or not the adjacent consecutive image sequences aresituated between adjacent similarity summary images.

It is determined that the consecutive image sequence is situated betweentwo similarity summary images when at least one image included in theconsecutive image sequence is situated between the two similaritysummary images. Specifically, all of the images included in theconsecutive image sequence need not necessarily be situated between thetwo similarity summary images.

When a given image included in the consecutive image sequence has beenextracted, and the extracted image is situated between two similaritysummary images, the consecutive image sequence from which the givenimage has been extracted may be subjected to the integrationdetermination process. In this case, the image selected as the objectsummary image is normally extracted from the consecutive image sequence.Note that another image included in the consecutive image sequence maybe extracted.

An example is described below with reference to FIGS. 8A and 8B. Theconsecutive image sequence B1 and the consecutive image sequence B2illustrated in FIG. 8A are situated between two adjacent similaritysummary images C1 and C2 illustrated in FIG. 8A according to the abovedefinition. Therefore, whether or not the consecutive image sequence B1and the consecutive image sequence B2 can be integrated is determined.

FIG. 9A illustrates a specific example of the consecutive image sequenceintegration determination process. The integration determination processis performed using the similarity summary image, and the object summaryimages selected from a plurality of processing target consecutive imagesequences. Specifically, the similarity summary image is deformed, andprojected onto each of a plurality of object summary images (in a numberat least equal to the number of integration determination targetconsecutive image sequences) to calculate the coverage area. Thisprocess may be performed based on the deformation information about theimages in the same manner as the first image summarization process thatutilizes the coverage ratio. Whether or not the target object detectedfrom each object summary image is situated within the coverage area isdetermined, and it is determined that the determination targetconsecutive image sequences can be integrated into the integratedconsecutive image sequence when the target object detected from eachobject summary image is situated within the coverage area. Since anidentical object range is captured within the similarity summary imageand each coverage area, it is likely that the target object is anidentical object when the target object detected from each objectsummary image is situated within the coverage area. In this case, it isunnecessary to extract the object summary images in a number at leastequal to the number of consecutive image sequences from a plurality ofconsecutive image sequences, and it suffices to integrate the pluralityof consecutive image sequences, and extract at least one object summaryimage from the integrated consecutive image sequence.

Note that the object summary image extracted from the integratedconsecutive image sequence (corresponding to the sum-set of theconsecutive image sequence B1 and the consecutive image sequence B2illustrated in FIG. 8A) coincides with one of the object summary imagesextracted from the consecutive image sequences before integration aslong as the selection reference feature quantity is not changed.Specifically, since the image B3 or B4 is extracted from the integratedconsecutive image sequence as the object summary image (see FIG. 8A),the consecutive image sequence integration process in a narrow sensecorresponds to a process that deletes the object summary image.

Since the similarity summary image is also present to follow the objectsummary image, a similar process is performed as illustrated in FIG. 9B.When it has been determined that the consecutive image sequences can beintegrated as a result of the process illustrated in FIG. 9A or 9B, theconsecutive image sequences are integrated.

FIG. 10 is a flowchart illustrating the integration process according tothe second embodiment. As illustrated in FIG. 10, a loop process (S401to S408) is performed during the integration process. The steps S402 toS407 are performed on all of the images of the input image sequence.

Specifically, whether or not the ith image of the input image sequenceis included in the first summary image sequence is determined (S402).When the ith image is included in the first summary image sequence (Yesin S402), the ith image is selected as the output summary image (S403).When the ith image is not included in the first summary image sequence(No in S402), whether or not the ith image is included in the secondsummary image sequence is determined (S404). When the ith image is notincluded in the second summary image sequence (No in S404), the ithimage is determined to be the deletion target image (S407). When the ithimage is included in the second summary image sequence (Yes in S404),the determination process based on the relationship with the similaritysummary image is performed on the ith image and the preceding objectsummary image (S405). Specifically, whether or not the ith image and thepreceding object summary image are situated between the adjacentsimilarity summary images is determined (see FIGS. 8A and 8B). When theith image and the preceding object summary image are not situatedbetween the adjacent similarity summary images (No in S405), theconsecutive image sequences are not integrated, and the ith image is notdeleted from the second summary image sequence. Therefore, the ith imageis selected as the output summary image in the step S403.

When the ith image and the preceding object summary image are situatedbetween the adjacent similarity summary images (Yes in S405) (i.e., whenit may be possible to integrate the consecutive image sequences), thedetermination process illustrated in FIGS. 9A and 9B is performed(S406). When the determination result in the step S406 is No (i.e., theconsecutive image sequences are not integrated), the ith image is notdeleted from the second summary image sequence. Therefore, the ith imageis selected as the output summary image in the step S403. When thedetermination result in the step S406 is Yes (i.e., the consecutiveimage sequences are integrated), the ith image is determined to be thedeletion target image in the step S407. The above process is performedon the entire input image sequence while incrementing the value i(initial value: i=0).

Note that the flowchart illustrated in FIG. 10 merely illustrates anexample of the process according to the second embodiment in that thenumber of consecutive image sequences that are integrated at a time islimited to two, and the object summary image corresponding to thebackward consecutive image sequence is deleted when the consecutiveimage sequences are integrated, for example. The process according tothe second embodiment may be implemented by a process that differs fromthe process illustrated in FIG. 10.

According to the second embodiment, the integration processing section300 integrates a plurality of consecutive image sequences that have beenset during the second image summarization process into one integratedconsecutive image sequence based on the first summary image sequence,and extracts at least one image from the integrated consecutive imagesequence as the summary image (object summary image) to update thesecond summary image sequence.

This makes it possible to update the second summary image sequence basedon the first summary image sequence. The second summary image sequenceis updated in a narrow sense by deleting an object summary image amongthe object summary images to reduce the number of images included in thesecond summary image sequence. The consecutive image sequences may beintegrated by performing the process illustrated in FIGS. 8A, 8B, and 9,or may be integrated using another method.

The integration processing section 300 may perform the integrationprocess on the first summary image sequence and the second summary imagesequence by selecting an image that is included in at least one of thefirst summary image sequence and the updated second summary imagesequence as the summary image (output summary image) of the outputsummary image sequence.

This makes it possible to perform the integration process on the firstsummary image sequence and the second summary image sequence using theupdated second summary image sequence. Therefore, it is possible toreduce the number of images included in the output summary imagesequence as compared with the case where the update process is notperformed (e.g., first embodiment) while achieving the advantages of theimage summarization process that utilizes the similarity and the imagesummarization process that utilizes the target object/scene recognitionprocess, and improve convenience to the user who utilizes the outputsummary image sequence, for example.

4. Third Embodiment

The third embodiment is described below. FIG. 11 illustrates a systemconfiguration example of an image summarization device according to thethird embodiment. As illustrated in FIG. 11, the basic configuration isthe same as that described above in connection with the firstembodiment, except that the first image summarization section 100 andthe integration processing section 300 are bidirectionally connected.

In the third embodiment, the integration processing section 300 acquiresthe results of the second image summarization process, and causes thefirst image summarization section 100 to perform the first imagesummarization process based on the acquired results.

FIGS. 12A to 12C illustrate a specific example of the above process. Thedescription given above in connection with FIGS. 8A and 8B also appliesto FIGS. 12A to 12C. As illustrated in FIG. 12A, the second summaryimage sequence is acquired by extracting at least one summary image fromthe consecutive image sequence based on the target object/scenerecognition process (see the first embodiment), for example. The firstimage summarization process is performed as illustrated in FIG. 12B (seethe first embodiment, for example). When the image S(i) has beenselected as the similarity summary image (first reference image), theimage S(i+1) (i.e., the next similarity summary image) is searched byperforming the determination process based on the coverage ratio aftersetting the second reference image.

However, an image that has been determined to be deleted based on thesimilarity (coverage ratio) should be selected as the output summaryimage when the target object (observation target) is captured within theimage, and the image represents the consecutive image sequence. Thefirst embodiment implements such an integration process. Therefore, thesame effects as described above can be expected to be achieved byselecting the similarity summary image (or the first reference image)during the first image summarization process using the results of thesecond image summarization process in addition to the similarity.

In the third embodiment, when searching the next similarity summaryimage after setting the similarity summary image (first referenceimage), an image that has been set to the object summary image isselected as the similarity summary image regardless of the similarity.As illustrated in FIG. 12B, the image indicated by E1 is determined tobe the deletion target image based on the similarity. However, the imageindicated by E1 has been selected as the object summary image (see D1 inFIG. 12A). Therefore, the image indicated by E1 is set to be thesimilarity summary image S(i+1) during the first image summarizationprocess according to the second embodiment.

The summary image may be set when a given condition has been satisfiedbased on the similarity, or when the summary image of the second summaryimage sequence has been found. In the example illustrated in FIG. 12C,the image S(i+2) has been selected based on the similarity, and theimage S(i+1) has been selected based on the object summary image. Theposition of the similarity summary image in a narrow sense (the resultof the first image summarization process according to the firstembodiment) in the image sequence is determined depending on anothersimilarity summary image (the preceding similarity summary image in anarrow sense). Therefore, when the similarity summary image has beenselected taking account of the results of the second image summarizationprocess, the images selected as the similarity summary image normallydiffer to a large extent as compared with the case where the results ofthe second image summarization process are not used.

Since the image sequence that takes account of both the targetobject/scene recognition process and the similarity can be acquired bythe first image summarization process based on the results of the secondimage summarization process, the output summary image sequence may begenerated using the results of the first image summarization process.

FIG. 13 is a flowchart illustrating the integration process according tothe third embodiment. Note that FIG. 13 actually illustrates the firstimage summarization process based on the results of the second imagesummarization process (see FIG. 4).

Steps S501 to S511 illustrated in FIG. 13 are performed in the samemanner as the steps S101 to S111 illustrated in FIG. 4, respectively,and detailed description thereof is omitted. In FIG. 13, steps S512 toS514 are added after the step S505 illustrated in FIG. 4. In the stepS512, whether or not the processing target ith image has been selectedas the object summary image as a result of the second imagesummarization process is determined. When it has been determined thatthe ith image has been selected as the object summary image as a resultof the second image summarization process (Yes in S512), the ith imageis set to be the next similarity summary image (S513), I_(S)is set to i(S514), and the similarity summary image set in the step S514 is set tobe the next first reference image (S502).

According to the third embodiment, the integration processing section300 performs the integration process on the first image summarizationprocess and the second image summarization process by causing the firstimage summarization section to perform the first image summarizationprocess based on the results of the second image summarization process.

According to this configuration, since the image summarization processbased on the determination as to the similarity between the images, andthe image summarization process based on the target object/scenerecognition process, can be integrated to acquire the output summaryimage sequence, it is possible to implement an image summarizationprocess that achieves the advantages of each image summarizationprocess.

The first image summarization section 100 may select the image (objectsummary image) included in the second summary image sequence from theplurality of images as the reference image based on the integrationprocess, select the determination target image from the plurality ofimages, calculate the coverage ratio of the determination target imageby the reference image as the similarity based on the deformationinformation about the reference image and the determination targetimage, and perform the first image summarization process that determineswhether or not the determination target image can be deleted based onthe coverage ratio.

This makes it possible to perform the reference image selection processduring the first image summarization process based on the results of thesecond image summarization process as the integration process on thefirst image summarization process and the second image summarizationprocess. More specifically, the object summary image included in thesecond summary image sequence may be selected as the reference image,and the image other than the object summary image may be processed basedon the similarity.

The integration processing section 300 may acquire the first summaryimage sequence generated by the first image summarization section as theoutput summary image sequence via the integration process.

This makes it possible to utilize the results of the first imagesummarization process based on the second image summarization process asthe output summary image sequence. According to the third embodiment, animage that has been selected as the summary image (object summary image)as a result of the second image summarization process is selected as thesummary image (similarity summary image) during the first imagesummarization process. Therefore, since the image based on the targetobject/scene recognition process is allowed to remain during the firstimage summarization process, it is unnecessary to take account of theintegration process on the first summary image sequence and the secondsummary image sequence, for example.

5. Fourth Embodiment

The fourth embodiment is described below. A system configuration exampleof an image summarization device according to the fourth embodiment isthe same as that illustrated in FIG. 11 (see the third embodiment), anddetailed description thereof is omitted. The fourth embodiment utilizesa method that combines the method according to the third embodiment withthe second summary image sequence update process (consecutive imagesequence integration process) according to the second embodiment.Specifically, the first image summarization process based on the resultsof the second image summarization process is performed in the samemanner as in the third embodiment, and whether or not the second summaryimage sequence can be updated is determined based on the acquired firstsummary image sequence. When the second summary image sequence can beupdated, the first image summarization process is performed based on theresults of the second image summarization process after the updateprocess, and the acquired new first summary image sequence is acquiredas the output summary image sequence.

FIGS. 14A to 14E illustrate a specific example of the above process.FIG. 14A illustrates the second summary image sequence that is acquiredfirst, and FIG. 14B illustrates the first summary image sequence whenthe results of the second image summarization process are not used.Since the first image summarization process is performed using theresults of the second image summarization process in the same manner asin the third embodiment, the first summary image sequence illustrated inFIG. 14C is acquired.

After the first summary image sequence illustrated in FIG. 14C has beenacquired, whether or not the second summary image sequence (see FIG.14A) can be updated is determined using the first summary imagesequence. Specifically, whether or not a plurality of consecutive imagesequences can be integrated into the integrated consecutive imagesequence is determined in the same manner as described above inconnection with the second embodiment. For example, the consecutiveimage sequences F1 and F2 illustrated in FIG. 14A are situated betweenadjacent similarity summary images G1 and G2, and subjected to theintegration determination process.

When the second summary image sequence has been updated as illustratedin FIG. 14D, the object summary image indicated by F3 in FIG. 14A isdeleted. Therefore, the image corresponding to the object summary imageindicated by F3 need not be allowed to remain in the output summaryimage. Specifically, since the image G1 included in the first summaryimage sequence illustrated in FIG. 14C is unnecessary, it is necessaryto change the first summary image sequence. In this case, the firstimage summarization process is performed again based on the updatedsecond summary image sequence illustrated in FIG. 14D, and the new firstsummary image sequence illustrated in FIG. 14E is acquired.

Note that the process may be performed on the entire input imagesequence when acquiring the first summary image sequence illustrated inFIG. 14E. However, since the deletion target image in FIG. 14C can bedetermined by updating the second summary image sequence, it can bedetermined that the images that precede the similarity summary image(G1) that precedes the deletion target image G1 do not change withoutperforming the first image summarization process again. Therefore, theprocess may be performed on only the images that follow the image G3illustrated in FIG. 14C.

FIG. 15 is a flowchart illustrating the integration process according tothe fourth embodiment. The second image summarization process isperformed (S601). The process in the step S601 corresponds to theprocess illustrated in FIG. 5. The first image summarization process isperformed based on the results (the second summary image sequence in anarrow sense) of the second image summarization process (S602). Theprocess in the step S602 corresponds to the process illustrated in FIG.13.

The second summary image sequence is updated based on the first summaryimage sequence (S603). The process in the step S603 corresponds to theprocess in the steps S404 to S406 illustrated in FIG. 10, for example.Whether or not the second summary image sequence has changed as a resultof the update process (whether or not a summary image among the summaryimages included in the second summary image sequence has been deleted ina narrow sense) is determined (S604). When the second summary imagesequence has changed as a result of the update process, the first imagesummarization process is performed based on the updated second summaryimage sequence (S605). After completion of the step S605, or when thesecond summary image sequence has not changed as a result of the updateprocess (No in S604), the corresponding first summary image sequence isset to be the output summary image sequence (S606), and the process isterminated. When the second summary image sequence has changed as aresult of the update process (Yes in S604), the processing results inthe step S605 are set to be the output summary image sequence. When thesecond summary image sequence has not changed as a result of the updateprocess (No in S604), the processing results in the step S602 are set tobe the output summary image sequence.

When the first image summarization process based on the results of thesecond image summarization process is referred to as “step A”, and thesecond summary image sequence update process based on the first summaryimage sequence is referred to as “step B”, the first step A(corresponding to the S602 illustrated in FIG. 15) and the step B (S603)using the results of the first step A are performed, and the second stepA (S605) is performed using the results of the step B. Note that it islikely that the second step B can be performed using the results of thesecond step A (i.e., the second summary image sequence changes as aresult of the update process) depending on the input image sequence.

As a modification of the fourth embodiment, the step A and the step Bmay be repeated an arbitrary number of times using the results of thepreceding step. In this case, the process may be terminated when thestep A has been performed N (N is an integer equal to or larger than 2)times, and the results may be set to be the output summary imagesequence. Alternatively, when it has been detected that the step B couldnot be performed (or when it has been detected that the second summaryimage sequence did not change although the step B was performed), theresults of the preceding the step A may be set to be the output summaryimage sequence.

According to the fourth embodiment, the integration processing section300 determines whether or not the second summary image sequence updateprocess that reduces the number of images included in the second summaryimage sequence can be performed based on the first summary imagesequence generated by the first image summarization section via theintegration process.

This makes it possible to determine whether or not the second summaryimage sequence update process (see the second embodiment) can beperformed using the first summary image sequence when using the methodaccording to the third embodiment (the first summary image sequence isset directly to be the output summary image sequence in the thirdembodiment). The integration process on the first image summarizationprocess and the second image summarization process has been performedwhen the first summary image sequence has been calculated (see the thirdembodiment). However, the results of the integration process do not takeaccount of a reduction in the number of images included in the outputsummary image sequence. Since convenience to the user can be improved byreducing the number of images included in the output summary imagesequence, it is advantageous to reduce the number of images included inthe output summary image sequence. The number of images included in theoutput summary image sequence can be reduced by the second summary imagesequence update process (e.g., consecutive image sequence integrationprocess) used in the second embodiment.

The integration processing section 300 may perform the second summaryimage sequence update process when it has been determined that thesecond summary image sequence update process can be performed. Theintegration processing section 300 may perform the integration processon the first image summarization process and the second imagesummarization process by causing the first image summarization sectionto perform the first image summarization process based on the results ofthe second image summarization process after the second summary imagesequence update process.

This makes it possible to perform the first image summarization processbased on the results (updated second summary image sequence) of thesecond image summarization process after the second summary imagesequence update process. The number of object summary images included inthe second summary image sequence can be reduced (from FIG. 14A to FIG.14D) by performing the second summary image sequence update process.However, the deletion process is not reflected directly in the firstsummary image sequence (FIG. 14 C) corresponding to the output summaryimage sequence. When the image F3 in FIG. 14 A has been deleted by theupdate process, it is undesirable to merely delete the correspondingimage (G1 in FIG. 14C) from the first summary image sequence. This isbecause the images situated between the image G3 and the image G2 cannotbe covered based on the similarity when the image H2 (or the image thatis closer to the image H3 than the image H2) is not selected as the nextsimilarity summary image that follows the image H3 (see FIG. 14B (i.e.,the results of the first image summarization process that does not takeaccount of the second image summarization process). However, when theimage G1 in FIG. 14C is deleted, the images situated between the imageG3 and the image G2 cannot be covered since the interval between theimage G3 and the image G2 is too long. Therefore, when the secondsummary image sequence update process has been performed (see FIGS. 14Aand 14D), it is desirable to perform the first image summarizationprocess again using the updated second summary image sequence to acquirethe first summary image sequence illustrated in FIG. 14E instead ofmerely deleting the image G1 (see FIG. 14C).

The integration processing section 300 may acquire the first summaryimage sequence generated by the first image summarization section as theoutput summary image sequence via the integration process.

This makes it possible to set the first summary image sequence acquiredby the first image summarization process based on the updated secondsummary image sequence to be the output summary image sequence. Sincethe image based on the target object/scene recognition process isallowed to remain during the first image summarization process in thesame manner as in the third embodiment, it is unnecessary to takeaccount of the integration process on the first summary image sequenceand the second summary image sequence, for example. The second summaryimage sequence may not change (i.e., the number of object summary imagescannot be reduced) even when the second summary image sequence updateprocess has been performed. In this case, even if the first imagesummarization process is performed based on the updated second summaryimage sequence, the output results are the same as those obtained by thefirst image summarization process based on the second summary imagesequence that is not updated. Therefore, it is desirable to skip thefirst image summarization process based on the updated second summaryimage sequence (see the flowchart illustrated in FIG. 15). In this case,the first summary image sequence based on the original second summaryimage sequence is acquired as the output summary image sequence (i.e.,the processing results in the step S602 in FIG. 15).

6. Fifth Embodiment

The fifth embodiment is described below. A system configuration exampleof an image processing device will be described first, the flow of theprocess will then be described using a flowchart, and the details of thefirst deletion determination process and the second deletiondetermination process will be described thereafter.

6.1 System Configuration Example

FIG. 17 illustrates a system configuration example of an imageprocessing device according to the fifth embodiment. The imageprocessing device includes a processing section 100, an image sequenceacquisition section 30, and a storage section 50.

The processing section 100 performs an image summarization process thatdeletes some of a plurality of images included in an image sequenceacquired by the image sequence acquisition section 30. The function ofthe processing section 100 may be implemented by hardware such as aprocessor (e.g., CPU) or an ASIC (e.g., gate array), a program, or thelike.

The image sequence acquisition section 30 acquires the image sequencethat is subjected to the image summarization process. The image sequenceacquired by the image sequence acquisition section 30 may include RGBchannel images that are arranged in time series. Alternatively, theimage sequence acquired by the image sequence acquisition section 30 maybe a spatially consecutive image sequence (e.g., an image sequence thatincludes spatially arranged images that have been captured using imagingdevices arranged in a row). Note that the images included in the imagesequence are not limited to RGB channel images. Another color space(e.g., gray channel image) may also be used.

The storage section 50 stores the image sequence acquired by the imagesequence acquisition section 30, and serves as a work area for theprocessing section 100 and the like. The function of the storage section50 may be implemented by a memory (e.g., RAM), a hard disk drive (HDD),or the like.

As illustrated in FIG. 17, the processing section 100 may include anattention image sequence setting section 1001, a first reference imageselection section 1002, a first determination target image selectionsection 1003, a first deletion determination section 1004, a partialimage sequence setting section 1005, a second reference image selectionsection 1006, a second determination target image selection section1007, and a second deletion determination section 1008. Note that theconfiguration of the processing section 100 is not limited to theconfiguration illustrated in FIG. 17. Various modifications may be made,such as omitting some of the elements illustrated in FIG. 17, or addingother elements. Note that each section illustrated in FIG. 17 isprovided to describe each subroutine when the image summarizationprocess performed by the processing section 100 is divided into aplurality of subroutines. The processing section 100 does notnecessarily include each section illustrated in FIG. 17 as an element.

The attention image sequence setting section 1001 extracts an attentionimage from a plurality of images included in the image sequence(hereinafter may be referred to as “acquired image sequence” in order toclearly distinguish the image sequence from the attention imagesequence, the summary image sequence, and the like) acquired by theimage sequence acquisition section 30, and sets an attention imagesequence that includes one extracted attention image or a plurality ofextracted attention images. The term “attention image” used hereinrefers to an image in which an attention area (e.g., lesion) iscaptured. The processing section 100 may detect the attention area. Forexample, the processing section 100 may perform given image processingon each image of the acquired image sequence to determine whether or notthe attention area has been captured, and determine the image in whichthe attention area is captured to be the attention image. The attentionarea may be detected in various ways. For example, the attention areamay be detected by extracting an edge component from the image, ordetermining the color or the like from the pixel value.

Note that the attention area need not necessarily be detected by theimage processing device. For example, the image sequence acquisitionsection 30 may acquire an image sequence in which metadata is added toeach image, the metadata indicating whether or not each image is theattention image. In this case, the attention image sequence settingsection 1001 does not perform the attention area detection process, andsets the attention image sequence based on a metadata readout process.

The first reference image selection section 1002 selects a firstreference image from the plurality of images included in the attentionimage sequence. The first determination target image selection section1003 selects an image among the plurality of images included in theacquired image sequence that differs from the first reference image as afirst determination target image. Note that the first determinationtarget image selection section 1003 selects the first determinationtarget image from images among the plurality of images included in theacquired image sequence that differ from the attention image.

The first deletion determination section 1004 determines whether or notthe first determination target image can be deleted based on theselected first reference image and the selected first determinationtarget image. The details thereof are described later.

The partial image sequence setting section 1005 sets a partial imagesequence that includes a plurality of images among the plurality ofimages included in the acquired image sequence based on the result ofthe first deletion determination process performed by the first deletiondetermination section 1004. The number of partial image sequences is notlimited to one. The partial image sequence setting section 1005 may seta plurality of partial image sequences. The details thereof aredescribed later.

When a plurality of partial image sequences have been set, the processperformed by the second reference image selection section 1006, theprocess performed by the second determination target image selectionsection 1007, and the process performed by the second deletiondetermination section 1008 are independently performed on each partialimage sequence. Specifically, a second reference image is selected fromthe plurality of images included in the partial image sequence. Thesecond determination target image selection section 1007 selects animage among the plurality of images included in the partial imagesequence that differs from the second reference image as a seconddetermination target image. The second deletion determination section1008 determines whether or not the second determination target image canbe deleted based on the selected second reference image and the selectedsecond determination target image. The details of each process aredescribed later.

6.2 Flow of Process

FIG. 18 is a flowchart illustrating the image summarization processaccording to the fifth embodiment. When the image summarization processhas started, the attention image is extracted from the plurality ofimages included in the acquired image sequence to set the attentionimage sequence (S701). The first reference image is selected from theattention image sequence (S702). When the process in the step S702 isperformed for the first time, the first image of the attention imagesequence may be selected as the first reference image. In the exampleillustrated in FIG. 16A, the attention image indicated by J1 is selectedas the first reference image. When the process in the step S702 isperformed subsequently, the first reference image is updated based onthe position of the current first reference image in the attention imagesequence. Specifically, the image of the attention image sequence thatimmediately follows the current first reference image may be selected asthe next first reference image. In the example illustrated in FIG. 16A,the image indicated by J2 is selected as the next first reference imagewhen the current first reference image is the image indicated by J1.

When the first reference image has been selected, the firstdetermination target image is selected from the acquired image sequence.When the process in the step S702 is performed for the first time, thefirst image among the images that are included in the acquired imagesequence and are not included in the attention image sequence isselected as the first determination target image. When the process inthe step S702 is performed subsequently, the first determination targetimage is updated based on the position of the current firstdetermination target image in the acquired image sequence. Specifically,the image among the images that are included in the acquired imagesequence and are not included in the attention image sequence thatimmediately follows the current first determination target image may beselected as the next first determination target image.

When the first reference image and the first determination target imagehave been selected, the first deletion determination process isperformed (S704). The first deletion determination process is performedbased on the coverage ratio. The details thereof are described later.After the step S704, information that indicates whether or not thecurrent first determination target image can be deleted is stored, andthe step S703 is performed. The images that are included in the acquiredimage sequence and are not included in the attention image sequence aresequentially selected as the first determination target image, andwhether or not each selected image can be deleted is determined byrepeating the steps S703 and S704.

When the step S704 has been performed on the last image among the imagesthat are included in the acquired image sequence and are not included inthe attention image sequence (i.e., the first determination target imagecannot be selected in the step S703), the step S702 is performed again.The first reference image is updated in the step S702. When the firstreference image has been updated, whether or not each image (each imageother than the images indicated by J1 to J3 in the example illustratedin FIG. 16A) among the images that are included in the acquired imagesequence and are not included in the attention image sequence can bedeleted is determined using the updated first reference image.

When the last image of the attention image sequence (image indicated byJ3 in the example illustrated in FIG. 16A) has been selected as thefirst reference image, and the process in the step S703 and the processin the step S704 using the first reference image have completed (i.e.,the first reference image cannot be selected in the step S702), thefirst deletion determination process is terminated, and the step S705 isperformed.

Whether or not each image that is included in the acquired imagesequence and is not included in the attention image sequence can bedeleted is determined by the above process. When a plurality ofattention images have been detected, whether or not each image can bedeleted is determined a plurality of times. Note that it is determinedthat an image that has been determined to be deleted at least once canbe deleted. This is because all of the attention images are allowed toremain in the summary image sequence, and no problem occurs when animage is covered by one of the attention images even if the image is notcovered by the remaining attention images.

It is thus determined that an image that has been determined to bedeleted is not allowed to remain in the summary image sequence. Notethat an image that has been determined to be allowed to remain in thesummary image sequence is not necessarily allowed to remain in thesummary image sequence, and the second deletion determination process isperformed on the image. This is because an image that is not covered byeach attention image is determined to be allowed to remain in thesummary image sequence, and no problem occurs when an image among theimages that have been determined to be allowed to remain in the summaryimage sequence is deleted provided that the image is covered by a givenimage.

All of the images that have been determined to be allowed to remain inthe summary image sequence by the first deletion determination processare not necessarily consecutive images. For example, when a deletiontarget interval based on the attention image has been determined by thefirst deletion determination process (see FIG. 16C), the images thatcannot be deleted are divided into three sections (see I2 to I4). Inthis case, it is inefficient to perform the deletion determinationprocess on each image that cannot be deleted. Specifically, since thefirst section and the second section are situated away from each otherin the acquired image sequence, it is likely that the imaging targetobject changed. Therefore, it is not likely that it is determined thatthe image included in the second section can be deleted based on theimage included in the first section. Therefore, it is normallyunnecessary to perform the process across a plurality of sections, andit suffices to perform the closed process on each section.

Therefore, an interval in which the images that have been determined tobe allowed to remain in the summary image sequence by the first deletiondetermination process are situated consecutively in the acquired imagesequence is detected, and a partial image sequence that includes theimages that correspond to the detected interval is set (S705). In theexample illustrated in FIG. 16D, three partial image sequences have beenset (see I5 to I7). When only one image has been determined to beallowed to remain in the summary image sequence by the first deletiondetermination process, the image is not set to be the partial imagesequence. Specifically, since the closed process is performed on eachpartial image sequence, it is impossible to determine whether or noteach image included in the partial image sequence can be deleted basedon another image when the partial image sequence includes only oneimage. Therefore, it is determined that one image that has beendetermined to be allowed to remain in the summary image sequence by thefirst deletion determination process and is not consecutive with anotherimage is allowed to remain in the summary image sequence.

When the partial image sequence has been set, the first image of thepartial image sequence is selected as the second reference image (S706).An image among the images included in the partial image sequence thatdiffers from the second reference image is selected as the seconddetermination target image (S707). When the process in the step S707 isperformed for the first time after the second reference image has beenset, the image that immediately follows the second reference image(i.e., the second image of the partial image sequence) is selected asthe second determination target image. When the process in the step S702is performed after the step S708, the second determination target imageis updated based on the position of the current second determinationtarget image in the partial image sequence. Specifically, the imageincluded in the partial image sequence that immediately follows thecurrent second determination target image may be selected as the nextsecond determination target image.

When the second reference image and the second determination targetimage have been selected, the second deletion determination process isperformed (S708). In the fifth embodiment, the determination processbased on the coverage ratio is performed in the same manner as the firstdeletion determination process. The details thereof are described later.

When it has been determined that the second determination target imagecan be deleted in the step S708, the second determination target imageis updated in the step S707. When the last image of the partial imagesequence has been selected as the second determination target image, andit has been determined that the second determination target image cannotbe deleted in the step S708 (i.e., all of the images of the partialimage sequence other than the second reference image are covered by thesecond reference image), it is determined that the second referenceimage is allowed to remain in the summary image sequence, and all of theimages of the partial image sequence other than the second referenceimage are deleted, and the process performed on the partial imagesequence is terminated. Specifically, the second determination targetimage cannot be selected in the step S707, and the step S705 isperformed again.

When it has been determined that at least one second determinationtarget image cannot be deleted, the second determination target imagemust be allowed to remain in the summary image sequence since the seconddetermination target image cannot be covered by the second referenceimage. Therefore, when it has been determined that the seconddetermination target image cannot be deleted in the step S708, an imagesequence that includes the current second determination target image andthe subsequent images in the partial image sequence is set to be a newpartial image sequence (S705). The processes in the steps S706 to S708are performed on the new partial image sequence to set the first imageof the new partial image sequence (i.e., the second determination targetimage that has been determined to be allowed to remain in the summaryimage sequence by the above process) to be the second reference image(i.e., the first image of the new partial image sequence is allowed toremain in the summary image sequence).

In the step S705, one partial image sequence or a plurality of partialimage sequences that have been set as a result of the first deletiondetermination process, and the partial image sequence that has been setas a result of the processes in the steps S706 to S708 performed on theone partial image sequence or the plurality of partial image sequencesare sequentially selected. When the process has been performed on all ofthe partial image sequences (i.e., when the partial image sequencecannot be selected in the step S705), the image summarization process isterminated. In the fifth embodiment, an image that has been set to bethe second reference image is allowed to remain in the summary imagesequence, and other images are deleted.

FIGS. 19A to 19D illustrate the flow of the process performed on oneimage sequence among a plurality of partial image sequences that havebeen set as a result of the first deletion determination process. Whenan image sequence that includes N images (see FIG. 19A) has been set tobe the partial image sequence as a result of the first deletiondetermination process, the first image is selected as the secondreference image, and the second image is selected as the seconddetermination target image. Whether or not the second determinationtarget image can be deleted is then determined.

When it has been determined that the second determination target imagecannot be deleted, the next second determination target image isselected. Specifically, the position of the second determination targetimage is shifted backward, and the third image is selected as the seconddetermination target image (see FIG. 19B). Whether or not the seconddetermination target image can be deleted is then determined, and theimage selected as the second determination target image is updated untilthe second determination target image that is determined to be allowedto remain in the summary image sequence is found.

When it has been determined that the second to (k−1)th images can bedeleted (i.e., the second to (k−1)th images are covered by the secondreference image to a certain extent), and the kth image cannot bedeleted (see FIG. 19C), the second to (k−1)th images are deleted (i.e.,the second to (k−1)th images are not allowed to remain in the summaryimage sequence). Since the kth image is not sufficiently covered by thesecond reference image, it is necessary to allow the kth image to remainin the summary image sequence. Therefore, the kth image and thesubsequent images (kth to Nth images) are set to be a new partial imagesequence.

The process illustrated in FIGS. 19A to 19C is then performed on the newpartial image sequence. Specifically, the process is performed on thenew partial image sequence that includes N−x+1 images (see FIG. 19D)using the first image (i.e., the kth image in FIG. 19C) as the secondreference image, and using the second image (i.e., the (k+1)th image inFIG. 19C) as the second determination target image. The subsequentprocess is performed in the same manner as described above. When it hasbeen determined that the second determination target image can bedeleted, the subsequent image is selected as the next seconddetermination target image. When it has been determined that the seconddetermination target image cannot be deleted, the second reference imageis allowed to remain in the summary image sequence, the image that canbe deleted is deleted, and the images that follow the current seconddetermination target image are set to be a new partial image sequence.The process is terminated when it has been determined that the lastimage of the partial image sequence can be deleted, or when only oneimage is included in the partial image sequence (i.e., when the seconddetermination target image cannot be selected).

Although FIG. 18 (flowchart) illustrates an example in which a pluralityof partial image sequences that have been set as a result of the firstdeletion determination process are sequentially processed one by one,the configuration is not limited thereto. When the configuration of theprocessing section 100 is suitable for parallel processing (e.g., when aCPU that includes a plurality of cores is used as the processing section100), or when the image processing device according to the fifthembodiment includes a plurality of computers, and distributed processingis performed by each computer, the second deletion determination processmay be performed on the plurality of partial image sequences inparallel. This makes it possible to reduce the time required for thesecond deletion determination process, for example.

6.3 First Deletion Determination Process

A process that utilizes the coverage ratio is described below as aspecific example of the first deletion determination process. Asillustrated in FIG. 20, the first deletion determination section 1004may include a deformation information acquisition section 1009, acoverage area calculation section 1010, a coverage ratio calculationsection 1011, and a threshold value determination section 1012. Notethat the configuration of the first deletion determination section 1004is not limited to the configuration illustrated in FIG. 20. Variousmodifications may be made, such as omitting some of the elementsillustrated in FIG. 20, or adding other elements.

The deformation information acquisition section 1009 acquires thedeformation information about two images. The details of the deformationinformation are the same as described above. The deformation informationacquisition section 1009 acquires the deformation information about thefirst reference image selected by the first reference image selectionsection 1002 and the first determination target image selected by thefirst determination target image selection section 1003.

The coverage area calculation section 1010 projects one of the twoimages onto the other image by utilizing the deformation information(deformation parameter) about the two images to calculate the coveragearea. The coverage ratio calculation section 1011 calculates thecoverage ratio based on the coverage area. The threshold valuedetermination section 1012 compares the calculated coverage ratio with agiven threshold value. The details of each process are the same asdescribed above in connection with the first embodiment, and detaileddescription thereof is omitted.

6.4 Second Deletion Determination Process

The second deletion determination process is described below. In thefifth embodiment, the second deletion determination process is alsoperformed based on the coverage ratio. As illustrated in FIG. 21, thesecond deletion determination section 1008 may include a deformationinformation acquisition section 1013, a coverage area calculationsection 1014, a coverage ratio calculation section 1015, and a thresholdvalue determination section 1016. Note that the configuration of thesecond deletion determination section 1008 is not limited to theconfiguration illustrated in FIG. 21. Various modifications may be made,such as omitting some of the elements illustrated in FIG. 21, or addingother elements.

The deformation information acquisition section 1013 acquires thedeformation information about the second reference image and the seconddetermination target image. The coverage area calculation section 1014deforms the second reference image based on the deformation informationabout the second reference image and the second determination targetimage, and projects the second reference image onto the seconddetermination target image to calculate the coverage area. The coverageratio calculation section 1015 calculates the coverage ratio from thearea ratio of the coverage area to the entire second determinationtarget image, for example. The threshold value determination section1016 compares the calculated coverage ratio with a given thresholdvalue. Note that the threshold value used for the second deletiondetermination process may differ from the threshold value used for thefirst deletion determination process.

In the fifth embodiment, the first deletion determination process andthe second deletion determination process are similar processes.Therefore, one deformation information acquisition section may beprovided instead of separately providing the deformation informationacquisition section 1009 and the deformation information acquisitionsection 1013. This also applies to the remaining sections. Specifically,the processing section 100 according to the fifth embodiment may includea deformation information acquisition section, a coverage areacalculation section, a coverage ratio calculation section, and athreshold value determination section, and the deformation informationacquisition section, the coverage area calculation section, the coverageratio calculation section, and the threshold value determination sectionmay implement both the first deletion determination process and thesecond deletion determination process.

According to the fifth embodiment, the image processing device includesthe image sequence acquisition section 30 that acquires an imagesequence that includes a plurality of images, and the processing section100 that performs the image summarization process that acquires thesummary image sequence based on the first deletion determination processand the second deletion determination process that delete some of theplurality of images included in the image sequence acquired by the imagesequence acquisition section 30 (see FIG. 17). The processing section100 sets the attention image sequence that includes one attention imageor a plurality of attention images included in the plurality of images.The processing section 100 selects the first reference image from theattention image sequence, selects the first determination target imagefrom the plurality of images, and performs the first deletiondetermination process that determines whether or not the firstdetermination target image can be deleted based on first deformationinformation that represents deformation between the first referenceimage and the first determination target image. The processing section100 sets the partial image sequence from the image sequence, a pluralityof images that have been determined to be allowed to remain by the firstdeletion determination process being consecutively arranged in thepartial image sequence. The processing section 100 selects the secondreference image and the second determination target image from thepartial image sequence, and performs the second deletion determinationprocess that determines whether or not the second determination targetimage can be deleted based on second deformation information thatrepresents deformation between the second reference image and the seconddetermination target image.

The attention image is an image that requires attention from the user.The attention image may be an image in which a specific object iscaptured, or may be an image having a specific color, for example.Whether or not each image is the attention image need not necessarily bedetermined from the image (e.g., by image processing). For example,sensor information from a sensor provided to the imaging device may beadded to each image as metadata, and whether or not each image is theattention image may be determined based on the metadata.

According to this configuration, since the image summarization processcan be performed from the viewpoint of whether or not each image is theattention image, and the viewpoint of whether or not the deletion targetimage is covered by the image that is allowed to remain based on thedeformation information about a plurality of images, it is possible toimplement an effective image summarization process. Note that the effectof reducing of the number of images is insufficient (see FIG. 16B) whenthe results of the processes that differ in viewpoint are merelycombined. According to the fifth embodiment, an efficient imagesummarization process can be implemented by utilizing the first deletiondetermination process that is performed based on the attention imagethat is allowed to remain, and the second deletion determination processthat is performed on the partial image sequence that cannot be deletedbased on the attention image. The first determination target image usedduring the first deletion determination process may be selected from theplurality of images included in the image sequence. Note that the firstdetermination target image may be selected from images among theplurality of images that are not included in the attention imagesequence taking account of the processing efficiency.

The processing section 100 may perform at least one of a first coverageratio determination process and a first structural element determinationprocess as the first deletion determination process. The processingsection 100 may perform at least one of a second coverage ratiodetermination process and a second structural element determinationprocess as the second deletion determination process. The first coverageratio determination process is a process that calculates the coverageratio of the first determination target image by the first referenceimage based on the first deformation information, and determines whetheror not the first determination target image can be deleted based on thecalculated coverage ratio. The first structural element determinationprocess is a process that determines whether or not the firstdetermination target image can be deleted based on the results of aprocess that utilizes a structural element that corresponds to theattention area and the first deformation information. The secondcoverage ratio determination process is a process that calculates thecoverage ratio of the second determination target image by the secondreference image based on the second deformation information, anddetermines whether or not the second determination target image can bedeleted based on the calculated coverage ratio. The second structuralelement determination process is a process that determines whether ornot the second determination target image can be deleted based on theresults of a process that utilizes a structural element that correspondsto the attention area and the second deformation information.

The term “attention area” used herein refers to an area for which theuser's observation priority is relatively higher than that of otherareas. For example, when the user is a doctor, and desires to performtreatment, the attention area refers to an area in which a mucousmembrane area or a lesion area is captured. If the doctor desires toobserve bubbles or feces, the attention area refers to an area in whicha bubble area or a feces area is captured. Specifically, the attentionarea for the user differs depending on the objective of observation, butis necessarily an area for which the user's observation priority isrelatively higher than that of other areas.

This makes it possible to perform at least one of the process thatutilizes the coverage ratio and the process that utilizes the structuralelement (see the second embodiment) as the process that utilizes thedeformation information. When using the coverage ratio, it is possibleto ensure that an area of a given image corresponding to a certain ratio(e.g., area ratio) is covered by the summary image that is allowed toremain in the summary image sequence even if the given image is deleted,and prevent a situation in which an area that cannot be observed occursdue to the image summarization process. When using the structuralelement, it is possible to ensure that at least part of an area capturedwithin a given image having a size corresponding to the structuralelement is captured within the summary image even if the given image isdeleted. Therefore, it is possible to prevent a situation in which anattention area that cannot be observed occurs due to the imagesummarization process by setting the structural element corresponding tothe attention area.

The processing section 100 may detect the attention area from theplurality of images, and set an image among the plurality of images inwhich the attention area has been detected to be the attention image.

This makes it possible to set the attention image based on the attentionarea. The attention area is normally similar to the attention area thatis used to set the structural element. Note that the attention area maybe set to differ from the attention area that is used to set thestructural element. For example, an area having a large amount of edgecomponents may be set to be the attention area for the attention image(e.g., folds, a blood vessel structure, and the like are extracted), anda lesion may be set to be the attention area for the structural element(e.g., a situation in which a lesion larger than a given size isprevented).

The image sequence acquisition section 30 may acquire a plurality of invivo images as the image sequence. The processing section 100 may detecta lesion area from the plurality of in vivo images as the attentionarea, and set an image among the plurality of in vivo images in whichthe lesion area has been detected to be the attention image.

According to this configuration, since the process can be performedusing the lesion area as the attention area, the process can be used fordiagnosis or the like that utilizes an image acquired by a capsuleendoscope or the like.

The fifth embodiment may be applied to an endoscope apparatus thatincludes an imaging section (e.g., an imaging section that is providedin an end section of the endoscope) and the above image processingdevice.

When a plurality of partial image sequences have been set, theprocessing section 100 may perform the second deletion determinationprocess on the plurality of partial image sequences in parallel.

According to this configuration, since the second deletion determinationprocess can be implemented by parallel processing, the processing speedcan be improved.

Note that part or most of the process performed by the image processingdevice and the like according to the fifth embodiment may be implementedby a program. In this case, the image processing device and the likeaccording to the fifth embodiment are implemented by causing a processor(e.g., CPU) to execute a program. Specifically, a program stored in aninformation storage medium is read, and executed by a processor (e.g.,CPU). The information storage medium (computer-readable medium) stores aprogram, data, and the like. The function of the information storagemedium may be implemented by an optical disk (e.g., DVD or CD), a harddisk drive (HDD), a memory (e.g., memory card or ROM), or the like. Theprocessor (e.g., CPU) performs various processes according to the firstembodiment based on a program (data) stored in the information storagemedium. Specifically, a program that causes a computer (i.e., a deviceincluding an operation section, a processing section, a storage section,and an output section) to function as each section according to thefifth embodiment (i.e., a program that causes a computer to execute theprocess implemented by each section) is stored in the informationstorage medium.

7. Sixth Embodiment

Another method that implements the first deletion determination processand the second deletion determination process is described below. Aconfiguration example of an image processing device according to thesixth embodiment is the same as that illustrated in FIG. 17, anddetailed description thereof is omitted. Note that the process performedby the first deletion determination section 1004 and the processperformed by the second deletion determination section 1008 differ fromthose described above. The flow of the process is the same as thatillustrated in FIG. 18 (flowchart), and detailed description thereof isomitted. Note that the process in the step S704 and the process in thestep S708 differ from those described above.

7.1 Deletion Determination that Utilizes Structural Element

A process that utilizes the structural element that corresponds to anattention area is described below as an example of the first deletiondetermination process and the second deletion determination process. Theattention area may or may not be the same as the attention area used todetermine the attention image by the attention image sequence settingsection 1001 (see FIG. 17). For example, when the attention imagesequence is set using an image in which a lesion is captured as theattention image, the structural element is also set based on the lesion.

When the attention area used to set the attention image sequence isidentical with the attention area used during the first deletiondetermination process, since an image in which the attention area iscaptured is included in the attention image sequence, and allowed toremain in the summary image sequence, it may be considered that it ismeaningless to determine the possibility that the attention area ismissed during the first deletion determination process. However, since alarge number of images that require the image summarization process areprocessed, the attention area is normally automatically detected by thesystem. In this case, it is difficult to detect the attention area withan accuracy of 100%, and an image may occur in which the attention areais captured, but cannot be detected (i.e., cannot be set to be theattention image). Therefore, it is considered that it is effective tomake a determination based on the possibility that the attention area ismissed in order to prevent a situation in which the attention areacaptured within such an image is missed, and it is advantageous to usean attention area similar to the attention area used to set theattention image sequence when setting the structural element (seebelow).

Note that the second deletion determination process is performed in thesame manner as the first deletion determination process, and detaileddescription thereof is omitted.

As illustrated in FIG. 22, the first deletion determination section 1004may include a structural element generation section 1017, a deformationinformation acquisition section 1009, a coverage area calculationsection 1010, and an attention area miss probability determinationsection 1018. Note that the configuration of the first deletiondetermination section 1004 is not limited to the configurationillustrated in FIG. 22. Various modifications may be made, such asomitting some of the elements illustrated in FIG. 22, or adding otherelements.

The structural element generation section 1017 generates the structuralelement used for the process performed by the attention area missprobability determination section 1018 based on the attention area. Forexample, an area having the same shape and the same size as those of theattention area is set to be the structural element. Note that thestructural element is not limited thereto.

The coverage area calculation section 1010 may calculate the coveragearea, and set an area of the second determination target image otherthan the coverage area to be a non-coverage area.

The attention area miss probability determination section 1018determines the probability that the attention area captured within thefirst determination target image is not observed (captured) within thefirst reference image (i.e., the attention area is missed) when thefirst determination target image is deleted.

A specific flow of the process is described below. The structuralelement generation section 1017 generates the structural element basedon the attention area. The structural element generation section 1017sets an area having a size and a shape that should not be missed to bethe structural element taking account of a typical size and the like ofthe attention area. For example, when the attention area is a lesion,and a lesion that is larger than a circle having a diameter of 30 pixelswithin the image is severe, and should not be missed, a circle having adiameter of 30 pixels is set to be the structural element.

When the first reference image and the first determination target imagehave been selected, the deformation information acquisition section 1009acquires the deformation information about the first reference image andthe first determination target image. The coverage area calculationsection 1010 projects the first reference image onto the firstdetermination target image by utilizing the acquired deformationinformation to calculate the coverage area.

When the coverage area has been calculated, the attention area missprobability determination section 1018 determines the probability thatthe attention area is missed. Specifically, the attention area missprobability determination section 1018 performs a erosion process thatutilizes the structural element on the non-coverage area of the firstdetermination target image other than the coverage area to determinewhether or not a residual area is present.

A specific example of the erosion process is described below withreference to FIGS. 23A to 23E. As illustrated in FIG. 23A, thenon-coverage area is necessarily a closed area, and the boundary of thenon-coverage area can be set. For example, an outer boundary BO1 and aninner boundary BO2 are set in FIG. 23A.

The erosion process that utilizes the structural element removes theoverlapping area of the non-coverage area and the structural elementwhen a reference point of the structural element is set at the boundaryof the non-coverage area. For example, when a circular area is set to bethe structural element, and the reference point of the structuralelement is the center of the circle, the erosion process draws a circleso that the center of the circle is situated at the boundary of thenon-coverage area, and excludes the overlapping area of the circle andthe non-coverage area from the non-coverage area. Specifically, a circleis drawn around a point situated at the outer boundary BO1 of thenon-coverage area (see FIG. 23A), and the overlapping area of the circleand the non-coverage area (i.e., the semicircular area indicated by thediagonal lines in FIG. 23A) is excluded from the non-coverage area.

Since the outer boundary BO1 is processed discretely, and includes aplurality of points, the above process may be performed on each pointamong the plurality of points. For example, a circle may be sequentiallydrawn around each point situated at the outer boundary BO1 in a givendirection (see FIG. 23A), and the overlapping area of each circle andthe non-coverage area may be excluded from the non-coverage area.

When part of the boundary of the non-coverage area coincides with theboundary of the determination target image, for example, thenon-coverage area may have only a single boundary. In such a case, theabove process may be performed on the single boundary. When thenon-coverage area has the outer boundary BO1 and the inner boundary BO2(see FIG. 23A), the above process is performed on the outer boundary BO1and the inner boundary BO2. Specifically, a circle is drawn around eachpoint situated at the inner boundary BO2 (see FIG. 23B), and theoverlapping area of each circle and the non-coverage area is excludedfrom the non-coverage area.

The non-coverage area is reduced through the erosion process. Forexample, the left part of the non-coverage area illustrated in FIG. 23Ais completely deleted (i.e., no residual area is present) by the erosionprocess performed on the outer boundary BO1 (see FIG. 23A) and theerosion process performed on the inner boundary BO2 (see FIG. 23B). Onthe other hand, a residual area RE that is not excluded by the erosionprocess performed on the outer boundary BO1 and the erosion processperformed on the inner boundary BO2 occurs in the lower right part ofthe non-coverage area (see FIG. 23C). Specifically, only the residualarea RE remains as a result of performing the erosion process thatutilizes the structural element over the entire non-coverage area (seeFIG. 23D).

The meaning of the erosion process when using a circle having a radius ras the structural element is discussed below. The non-coverage area(i.e., closed area) is considered to be an area that is surrounded by aboundary (different boundaries (e.g., BO1 and BO2) or a singleboundary). When the erosion process is performed on the boundary, apoint among the points included in the non-coverage area that issituated at a distance equal to or shorter than r from each pointsituated at the boundary is determined to be the deletion target.Specifically, the distance from the point included in the residual area(that is excluded from the deletion target) to an arbitrary pointsituated at the boundary is longer than r. Therefore, a circle having aradius r that is drawn around an arbitrary point within the residualarea does not intersect each boundary. This means that the entirety ofthe attention area represented by a circle having a radius R (=r) thatis drawn around a point within the residual area is included within thenon-coverage area. Note that the above basic idea is also applied evenwhen the structural element has a shape (e.g., quadrangle) other than acircle.

Specifically, when the residual area is present, an area thatcorresponds to the structural element is included within thenon-coverage area (see the lower right part in FIG. 23E). When theattention area (e.g., lesion) is situated at such a position, and thefirst determination target image is deleted, it is likely that theattention area cannot be observed even if the first reference image isallowed to remain. When the residual area is not present, at least partof the attention area is included within the coverage area (see theupper left part in FIG. 23E). In this case, at least part of theattention area remains in the first reference image even if the firstdetermination target image is deleted.

Therefore, the attention area miss probability determination section1018 performs the erosion process that utilizes the structural elementon the non-coverage area, and determines whether or not the firstdetermination target image can be deleted based on whether or not theresidual area is present.

7.2 Modification of Deletion Determination

The first deletion determination process and the second deletiondetermination process may be implemented by the process that utilizesthe coverage ratio or the process that utilizes the structural element,as described above. Note that the first deletion determination processand the second deletion determination process need not necessarily beimplemented by independently using the process that utilizes thecoverage ratio or the process that utilizes the structural element. Theprocess that utilizes the coverage ratio and the process that utilizesthe structural element may be used in combination.

For example, the first deletion determination process may be implementedby performing both the process that utilizes the coverage ratio and theprocess that utilizes the structural element, and the second deletiondetermination process may also be implemented by performing both theprocess that utilizes the coverage ratio and the process that utilizesthe structural element. In this case, it may be determined that thefirst determination target image can be deleted when it has beendetermined that the first determination target image can be deleted bythe determination process based on the coverage ratio and thedetermination process based on the structural element in order toprevent a situation in which an area that cannot be observed occurs, andprevent a situation in which the attention area is missed to improve theutility of the summary image sequence. Note that the threshold valuethat is compared with the coverage ratio during the first deletiondetermination process may or may not be identical with the thresholdvalue that is compared with the coverage ratio during the seconddeletion determination process. The structural element (the size thereofin a narrow sense) used for the first deletion determination process mayor may not be identical with the structural element (the size thereof ina narrow sense) used for the second deletion determination process.

The first deletion determination process and the second deletiondetermination process may be implemented by a different process. Forexample, the first deletion determination process may be implemented bythe process based on the coverage ratio, and the second deletiondetermination process may be implemented by the process based on thecoverage ratio and the process based on the structural element. In thiscase, since at least one of the first deletion determination process andthe second deletion determination process is performed from a pluralityof viewpoints (i.e., the coverage ratio and the structural element inthe above example), it is expected that the determination accuracy isimproved as compared with a process based on a single viewpoint.Moreover, since a process from a given viewpoint (i.e., the structuralelement in the above example) can be omitted during one of the firstdeletion determination process and the second deletion determinationprocess, the processing load can be reduced as compared with the casewhere both the first deletion determination process and the seconddeletion determination process are performed from a plurality ofviewpoints.

When using two viewpoints, it is desirable that at least one of thefirst deletion determination process and the second deletiondetermination process be performed based on the two viewpoints. Forexample, it is desirable to avoid a situation in which the firstdeletion determination process utilizes the coverage ratio, and thesecond deletion determination process utilizes the structural element.Specifically, sufficient accuracy may not be achieved depending on theprocessing target image when only the coverage ratio or the structuralelement is used. The determination accuracy is improved using both thecoverage ratio and the structural element when both the process thatutilizes the coverage ratio and the process that utilizes the structuralelement are performed on a combination of the reference image and thedetermination target image. However, a combination of the firstreference image and the first determination target image during thefirst deletion determination process does not overlap a combination ofthe second reference image and the second determination target imageduring the second deletion determination process, taking account of theabove selection method. Specifically, when the first deletiondetermination process utilizes the coverage ratio, and the seconddeletion determination process utilizes the structural element, thecoverage ratio and the structural element are used independentlyalthough the image summarization process utilizes the coverage ratio andthe structural element, and the determination accuracy may not besufficiently improved. In such a case, however, since the processingsection 100 must perform a plurality of different deletion determinationprocesses, the system configuration efficiency decreases.

Note that the deletion determination process may be performed from threeor more viewpoints using an element other than the coverage ratio andthe structural element. In this case, it is desirable that at least oneof the first deletion determination process and the second deletiondetermination process be performed using all of the three or moreviewpoints.

According to the sixth embodiment, the processing section 100 mayperform a second coverage ratio determination process as the seconddeletion determination process when performing a first coverage ratiodetermination process as the first deletion determination process. Theprocessing section 100 may perform a second structural elementdetermination process as the second deletion determination process whenperforming a first structural element determination process as the firstdeletion determination process.

This makes it possible to perform a determination process based on thecoverage ratio as the first deletion determination process and thesecond deletion determination process. It is also possible to perform adetermination process based on the structural element as the firstdeletion determination process and the second deletion determinationprocess. In this case, since the first deletion determination processand the second deletion determination process can be implemented from asingle viewpoint, the processing load can be reduced as compared withthe case of using both the coverage ratio and the structural element.

The processing section 100 may perform both the first coverage ratiodetermination process and the first structural element determinationprocess as the first deletion determination process. The processingsection 100 may perform both the second coverage ratio determinationprocess and the second structural element determination process as thesecond deletion determination process.

According to this configuration, since at least one of the firstdeletion determination process and the second deletion determinationprocess is implemented by both the coverage ratio determination processand the structural element determination process, the determinationaccuracy can be improved as compared with the case where both the firstdeletion determination process and the second deletion determinationprocess are implemented by the coverage ratio determination process, orboth the first deletion determination process and the second deletiondetermination process are implemented by the structural elementdetermination process. Note that both the coverage ratio determinationprocess and the structural element determination process may beperformed during both the first deletion determination process and thesecond deletion determination process. However, the processing load canbe reduced by simplifying (e.g., omitting the structural elementdetermination process) one of the first deletion determination processand the second deletion determination process.

The first coverage ratio determination process may be a determinationprocess based on the result of a comparison between a value thatrepresents the coverage ratio of the first determination target image bythe first reference image and a first coverage ratio threshold value.The first structural element determination process may be a process thatsets an element having a first size to be the structural element, andperforms the erosion process that utilizes the set structural element,or determines whether or not the set structural element is included inan area in which the first determination target image is not covered bythe first reference image.

The second coverage ratio determination process may be a determinationprocess based on the result of a comparison between a value thatrepresents the coverage ratio of the second determination target imageby the second reference image and a second coverage ratio thresholdvalue. The second structural element determination process may be aprocess that sets an element having a second size to be the structuralelement, and performs the erosion process that utilizes the setstructural element, or determines whether or not the set structuralelement is included in an area in which the second determination targetimage is not covered by the second reference image.

This makes it possible to perform a process that compares the calculatedcoverage ratio and the threshold value as the determination processbased on the coverage ratio. When the coverage ratio is calculated asillustrated in FIG. 2, it suffices that the determination processcompare the calculated coverage ratio with the threshold value.Therefore, the process is easy. It is also possible to perform theerosion process that utilizes the structural element (see FIGS. 23A to23E) as the determination process based on the structural element. Notethat the target of the erosion process that utilizes the structuralelement is not limited to the non-coverage area.

For example, the erosion process that utilizes the structural elementmay be performed on the determination target image (see FIG. 24A). Inthis case, the coverage-requiring area that must be covered by thereference image remains by setting the structural element so that theentirety of the attention area is not included within the area that isremoved by the erosion process (e.g., setting an element having a sizetwice that of the attention area as the structural element).Specifically, whether or not the determination target image can bedeleted may be determined based on whether or not the entirety of thecoverage-requiring area is covered by the reference image. Morespecifically, one of the reference image and the coverage-requiring areamay be deformed using the deformation information, and the inclusiondetermination process may be performed using the deformed area (seeFIGS. 25A and 25B). The determination target image can be deleted whenthe coverage-requiring area is included in the reference image, andcannot be deleted when the entirety of the coverage-requiring area isnot included in the reference image.

The deletion determination process that utilizes the structural elementis not limited to the deletion determination process that utilizes theerosion process. It suffices that the deletion determination processthat utilizes the structural element determine whether or not thestructural element is included in the non-coverage area. For example,the deletion determination process that utilizes the structural elementmay be implemented using a simple method that calculates a value thatcorresponds to the maximum size (diameter) of the non-coverage areabased on the distance (e.g., k1 to k6) from the point (e.g., p1 to p6)at the boundary of the coverage area to the boundary of thedetermination target image, or the distance from the point at theboundary of the determination target image to the boundary of thecoverage area, and compares the calculated value with the minimum sizeof the structural element (e.g., a structural element having the size asthat of the attention area) (see FIGS, 26A and 26B).

The processing section 100 may set a value that differs from the firstcoverage ratio threshold value to be the second coverage ratio thresholdvalue. The processing section 100 may set a size that differs from thefirst size to be the second size.

This makes it possible to change the determination reference valuecorresponding to the first deletion determination process and the seconddeletion determination process even when a process from a singleviewpoint is used, and implement a flexible two-step determinationprocess.

8. Seventh Embodiment

The fifth embodiment has been described above taking the methodillustrated in FIGS. 19A to 19D as an example of the method that selectsthe second reference image and the second determination target imageduring the second deletion determination process. Note that the methodthat selects the second reference image and the second determinationtarget image is not limited to the method illustrated in FIGS. 19A to19D. The seventh embodiment illustrates a method that sets two images(forward reference image and backward reference image) to be the secondreference image, and sets an image between the two second referenceimages to be the second determination target image.

In this case, the coverage area may be an area that corresponds to thesum-set of an area calculated by deforming the forward reference imagebased on the deformation information about the forward reference imageand the second determination target image, and an area calculated bydeforming the backward reference image based on the deformationinformation about the backward reference image and the seconddetermination target image (see FIG. 2). Specifically, no problem occurseven if the second determination target image is deleted when the seconddetermination target image is covered by at least one of the forwardreference image and the backward reference image. The process performedafter calculating the coverage area is the same as described aboveirrespective of whether the coverage ratio or the structural element isused.

When it has been determined that all of the images situated between theforward reference image and the backward reference images can bedeleted, all of the images situated between the forward reference imageand the backward reference images may be deleted as long as the forwardreference image and the backward reference image are allowed to remainin the summary image sequence. However, it is desirable to set theforward reference image and the backward reference image at positionssituated away from each other as much as possible while the conditionwhereby all of the images situated between the forward reference imageand the backward reference images can be deleted is satisfied, in orderto improve the effect of reducing the number of images through the imagesummarization process. Therefore, an optimum position is searched whilefixing the forward reference image, and changing the position of thebackward reference image. Specifically, the method illustrated in FIGS.3A and 1B is used.

The expression “the qth image is OK” is used when the qth image has beenselected as the backward reference image, and it has been determined bythe deletion determination process that all of the images situatedbetween the forward reference image and backward reference image can bedeleted, and the expression “the qth image is NG” is used when the qthimage has been selected as the backward reference image, and it has beendetermined by the deletion determination process that at least one ofthe images situated between the forward reference image and backwardreference image cannot be deleted, for convenience of description.

When the first to Nth images have been input as the partial imagesequence, the first image has been selected as the forward referenceimage, and the qth image has been selected as the backward referenceimage to search an optimum position of the backward reference image, thesecond to (q−1)th images are sequentially selected as the seconddetermination target image, and whether the qth image is OK or NG isdetermined. When the qth image is OK (i.e., when the interval betweenthe forward reference image and the backward reference image can beincreased), the next backward reference image is selected from the(q+1)th image and the subsequent images. When the qth image is NG (i.e.,when the interval between the forward reference image and the backwardreference image is too large), the image that precedes the qth image isbasically selected as the next backward reference image.

Specifically, the next summary image that follows the forward referenceimage is searched by updating the backward reference image with thesubsequent image when the qth image is OK, and updating the backwardreference image with the preceding image when the qth image is NG, untilthe end condition is satisfied. The number of images selected as thebackward reference image until the next summary image is found can bereduced while reducing the amount of calculations by appropriatelyupdating the position of the next backward reference image. The methodaccording to the seventh embodiment is described in detail below.

In the seventh embodiment, the second reference image selection section1006 includes a forward reference image selection section 1019 and abackward reference image selection section 1020 (see FIG. 27). Theforward reference image selection section 1019 selects the forwardreference image, and the backward reference image selection section 1020selects the backward reference image.

For example, the first image of the partial image sequence is selectedas the forward reference image. When the partial image sequence is apartial image sequence acquired by the first deletion determinationprocess (i.e., when the forward reference image selection process isperformed for the first time), an image other than the first image maybe selected as the forward reference image. Note that the followingdescription is given on the assumption that the forward reference imageis the first image of the partial image sequence unless otherwisespecified.

The backward reference image is then selected. For example, a backwardreference image selection interval is set that corresponds to the imagesfrom which the backward reference image is selected (corresponding tothe range in which the next summary image that follows the forwardreference image is searched). A semi-open interval [i, j) correspondingto the ith to jth images is set to be the backward reference imageselection interval. i corresponds to the image that immediately followsthe forward reference image (i=2 in a narrow sense), and j is set toN+2. j is set to N+2 since a virtual (N+1)th image can be set to be thebackward reference image. When the backward reference image is the(N+1)th image, all of the subsequent images can be covered by theforward reference image, and whether or not the backward reference imageis unnecessary is determined.

The backward reference image is selected from the backward referenceimage selection interval. The backward reference image is determinedbased on a given condition in order to efficiently perform the process.Specifically, when the backward reference image is selected for thefirst time after the forward reference image has been set, the (i+1)thimage (third image in a narrow sense) is selected as the backwardreference image.

FIG. 28A illustrates the process described above. FIG. 28A illustratesan image sequence in which N=12. The forward reference image is thefirst image, the backward reference image selection interval correspondsto the second to fourteenth images (i=2, j=14), and the backwardreference image is the third image.

After the backward reference image has been selected, the seconddetermination target image selection process, the coverage ratiocalculation process, and the deletion determination process areperformed (repeated) in the same manner as described above (detaileddescription thereof is omitted). In the example illustrated in FIG. 28A,it suffices to select the second image as the second determinationtarget image.

When a given image (the third image during the first process) has beenselected as the backward reference image, and the given image is OK(i.e., the position of the backward reference image can be furthershifted away from the forward reference image), the image that followsthe current backward reference image is selected as the next backwardreference image.

For example, when the current backward reference image is the ath imagefrom the forward reference image, the (2×a)th image from the forwardreference image may be selected as the next backward reference image.Specifically, when the third image (i.e., the second image from theforward reference image) has been selected as the backward referenceimage, and the third image is OK, the fifth image (i.e., the fourthimage from the forward reference image) is selected as the next backwardreference image (see FIG. 28B).

When the qth image is OK, it is unnecessary to select the (q−1)th imageand the images that precede the (q−1)th image as the summary image. Inthis case, the backward reference image selection interval may beupdated since no advantage is obtained even if the image that precedesthe qth image is selected as the backward reference image. Specifically,the starting point i of the backward reference image selection intervalmay be set to i=q. Since the backward reference image is selected fromthe backward reference image selection interval, the image that precedesthe current backward reference image is not selected when the startingpoint i is set to i=q. For example, when the third image is OK (i.e.,when the second image is not selected as the summary image), the secondimage is excluded from the backward reference image selection interval,and the starting point of the backward reference image selectioninterval is updated with the third image (see FIG. 28B).

Likewise, when the fifth image is OK (i.e., the second to fourth imagesare selected as the second determination target image, and the seconddeletion determination process is performed), the ninth image isselected as the next backward reference image, and the starting point ofthe backward reference image selection interval is updated with thefifth image (see FIG. 28C).

However, when the qth image has been selected as the backward referenceimage, and the qth image is OK, it is likely that an image that issituated significantly away from the qth image is selected as the nextbackward reference image as the value q increases (see FIG. 28C). Forexample, a situation may occur in which an image that follows the(N+1)th image may be set to be a candidate for the backward referenceimage (i.e., the backward reference image cannot be selected), or theinterval between the current backward reference image and the nextbackward reference image increases to a large extent, and the nextsummary image search process becomes inefficient.

Therefore, another method may be used in combination with the abovemethod when selecting an image that follows the current backwardreference image as the next backward reference image. For example, thenext backward reference image may be determined based on the value(q+j)/2. For example, when the ninth image is OK, the starting point ofthe backward reference image selection interval is updated with theninth image (i.e., the backward reference image selection interval is asemi-open interval 119, 14)). Specifically, the center of the searchrange can be set to be the processing target by selecting an imagearound the center of the search range as the next backward referenceimage. The method that halves the search range by determining the centerof the search range is a widely known binary search method, and it isknown that the binary search method is advantageous from the viewpointof the amount of calculations. The binary search method can be appliedto the backward reference image selection interval since all of theimages that precede a given image are determined to be OK when the givenimage is OK, and all of the images that follow a given image aredetermined to be NG when the given image is NG. Specifically, it isconsidered that an efficient process can be implemented by selecting thenext backward reference image from approximately the center pointbetween the current backward reference image and the end point of thebackward reference image selection interval.

A method that doubles the distance from the forward reference image, anda method that corresponds to the binary search method may be used incombination. For example, when the qth image is the current backwardreference image, the kth image that satisfies the following expression(1) may be selected as the next backward reference image. Note thatmin(a, b) outputs the smaller of a and b.

$\begin{matrix}{k = {\min \left( {{{2\; q} - 1},\frac{q + j}{2}} \right)}} & (1)\end{matrix}$

When the qth image is NG, the next backward reference image is selectedfrom the images that precede the current backward reference image. Thenext backward reference image may be determined using various methods.For example, the next backward reference image may be determined using amethod that corresponds to the binary search method. In this case, sincethe starting point of the backward reference image selection interval isthe ith image, the next backward reference image is determined based onthe value (i+q)/2. Since the qth image is NG, the qth image and theimages that follow the qth image are not selected as the summary image.Therefore, the end point of the backward reference image selectioninterval may be updated (i.e., j=q). FIG. 28D illustrates an examplewhen the ninth image is NG. The seventh image is selected as the nextbackward reference image, and the end point j of the backward referenceimage selection interval is updated with j=9.

Note that a semi-open interval is used as the backward reference imageselection interval for convenience of explanation. Specifically, sincethe qth image may be selected as the summary image when the qth image isOK, it is desirable that the starting point i (i=q) of the backwardreference image selection interval be included in the backward referenceimage selection interval. Since the qth image is not selected as thesummary image when the qth image is NG, it is desirable that the endpoint j (j=q) of the backward reference image selection interval not beincluded in the backward reference image selection interval. Therefore,the backward reference image selection interval is represented by [i,j). The backward reference image selection interval may be representedby an open interval or a closed interval depending on the sign or theexpression.

The backward reference image selection interval (i.e., the next summaryimage search range in a narrow sense) is narrowed by the above process.Since the next summary image is the kth image when the kth image is OKand the (k+1)th image is NG, the process is terminated when an imagethat is OK and an image that is NG are adjacent to each other. In theabove example, it is considered that the process is performed in abinary search manner immediately before the process is terminated (seeFIG. 28E). In FIG. 28E, the ith image is OK, the jth image is NG, andthe qth image between the ith image and the jth image is selected as thebackward reference image. FIG. 28F illustrates the case where the qthimage is OK, and FIG. 28G illustrates the case where the qth image isNG. In FIGS. 28F and 28G, the starting point and the end point of thebackward reference image selection interval are adjacent to each other,the image corresponding to the starting point is OK, and the imagecorresponding to the end point is NG. In this case, the imagecorresponding to the starting point is selected as the next summaryimage, and the search process performed on the partial image sequence isterminated.

When the next summary image has been found, an image sequence thatincludes the next summary image and the images that follow the nextsummary image is set to be a new partial image sequence. The subsequentprocess is performed in the same manner as described above, and detaileddescription thereof is omitted.

FIG. 29 is a flowchart illustrating the image summarization processaccording to the seventh embodiment. Steps S801 to S805 are performed inthe same manner as the steps S701 to S705 illustrated in FIG. 18,respectively, and detailed description thereof is omitted. When thepartial image sequence has been set, the first image of the processingtarget partial image sequence is selected as the forward reference image(S806), and the backward reference image selection interval is set(S807). When the step S807 is performed immediately after the step S806,a semi-open interval i, j) that satisfies i=2 and j=N+2 may be set, forexample. When the step S807 is performed after the step S810 or S811,the backward reference image selection interval is updated.

When the backward reference image selection interval has been set (orupdated) in the step S807, whether or not the starting point and the endpoint of the backward reference image selection interval are adjacent toeach other (i.e., whether or not j=i+1 is satisfied) is determined(S808). When it has been determined that the starting point and the endpoint of the backward reference image selection interval are adjacent toeach other in the step S808 (i.e., when it has been determined that theith image is the next summary image that follows the first image(forward reference image)), the ith image and the subsequent images areset to be the partial image sequence in the step S805.

When it has been determined that the starting point and the end point ofthe backward reference image selection interval are not adjacent to eachother in the step S808 (i.e., when the next summary image has not beenfound), the backward reference image is selected from the backwardreference image selection interval set in the step S807 (S809). When theprocess in the step S809 is performed for the first time after theforward reference image has been set in the step S806, the (i+1)th image(i.e., the image that follows the forward reference image at an intervalof one image) may be selected, for example. When the process in the stepS809 is not performed for the first time after the forward referenceimage has been set in the step S806, the next backward reference imageis selected corresponding to the position of the current forwardreference image.

After the backward reference image has been selected in the step S809,the second determination target image is selected (S810). When theprocess in the step S810 is performed for the first time after thebackward reference image has been selected in the step S809, the firstimage (the second image in FIG. 28A) among the images situated betweenthe forward reference image and the backward reference image isselected. The second deletion determination process (e.g., coverage areacalculation process, coverage ratio calculation process, and thresholdvalue determination process) after the second determination target imagehas been selected is performed in the same manner as in the step S708illustrated in FIG. 18. When it has been determined that the seconddetermination target image can be deleted in the step S811, the seconddetermination target image is updated with the image that immediatelyfollows the current second determination target image (S809), and theprocess is performed in the same manner as described above. Whether ornot all of the images situated between the forward reference image andthe backward reference image can be deleted, or at least one of theimages situated between the forward reference image and the backwardreference image cannot be deleted, is determined by repeating the stepsS810 and S811. When it has been determined that all of the imagessituated between the forward reference image and the backward referenceimage can be deleted (second determination target image=backwardreference image), the step S807 is performed again. When it has beendetermined that at least one of the images situated between the forwardreference image and the backward reference image cannot be deleted, itis determined that the second determination target image cannot bedeleted in the step S811, and the step S807 is performed again. It isnecessary to store information that represents whether the step S807 isperformed after the step S810 or S811, and change the process in thestep S807 based on the information (not illustrated in FIG. 29).

When the step S807 is performed after the step S810 (i.e., when all ofthe images can be deleted), the starting point of the backward referenceimage selection interval is updated, and the image that follows thecurrent backward reference image is selected as the next backwardreference image in the step S807. When the step S807 is performed afterthe step S811 (i.e., when at least one of the images cannot be deleted),the end point of the backward reference image selection interval isupdated, and the image that precedes the current backward referenceimage is selected as the next backward reference image in the step S807.

According to the seventh embodiment, when the first to Nth (N is aninteger equal to or larger than 2) images have been set to be thepartial image sequence, the processing section 100 selects the forwardreference image and the backward reference image as the second referenceimage, the forward reference image being the pth (p is an integer thatsatisfies 1≦p≦N) image, and the backward reference image being the qth(q is an integer equal to or larger than p+2) image. The processingsection 100 selects the rth (r is an integer that satisfies p+1≦r≦q−1)image as the second determination target image. The processing section100 calculates forward deformation information that representsdeformation between the forward reference image and the seconddetermination target image, and backward deformation information thatrepresents deformation between the backward reference image and thesecond determination target image, as the second deformationinformation, and determines whether or not the second determinationtarget image can be deleted based on the calculated forward deformationinformation and the calculated backward deformation information.

This makes it possible to set the forward reference image and thebackward reference image during the second deletion determinationprocess. The process according to the seventh embodiment that utilizesthe deformation information basically aims to ensure that the image thatis deleted is covered by the image that is allowed to remain (i.e., thearea ratio or the like is high when using the coverage ratio, and atleast part of the attention area is observed (captured) when using thestructural element). Therefore, no problem occurs when the determinationtarget image is deleted provided that the determination target imagecannot be covered by one of a plurality of images that are allowed toremain, but can be covered by combining the plurality of images that areallowed to remain. According to the seventh embodiment, the probabilitythat it is determined that the determination target image can be deletedis increased by utilizing two reference images to improve the effect ofreducing the number of images due to the image summarization process.

The processing section 100 may select the backward reference image fromthe backward reference image selection interval in which the startingpoint and the end point are set corresponding to the (p+2)th to Nthimages, and determine whether or not the second determination targetimage can be deleted based on the forward reference image and thebackward reference image. The processing section 100 may select the xth(x is an integer that satisfies x>q) image included in the backwardreference image selection interval as the next backward reference image,and update the starting point of the backward reference image selectioninterval with the qth image when it has been determined that the (p+1)thto (q−1)th images can be deleted.

The backward reference image selection interval includes the (p+2)th toNth images that are candidates for the backward reference image.However, since a virtual image (e.g., (N+1)th image) can be selected asthe backward reference image, the end point of the backward referenceimage selection interval may be larger than N. Since the backwardreference image selection interval is used as the next summary image(i.e., the next summary image that follows the forward reference imagethat has been determined to be the summary image) search range, an imagethat is not selected as the backward reference image, but may beselected as the summary image, may be included in the backward referenceimage selection interval. In this case, the image ((p+1)th image) thatimmediately follows the forward reference image may be set to be thestarting point of the backward reference image selection interval.

This makes it possible to flexibly determine the position of the nextbackward reference image when updating the backward reference image. Thesearch range may be narrowed by thoroughly checking the search rangefrom the first image (e.g., by sequentially updating the backwardreference image with the image that immediately follows the currentbackward reference image). Alternatively, the search range may besignificantly narrowed by the unit determination that determines whetherthe qth image is OK or NG by allowing a non-adjacent image to beselected as the next backward reference image. An effective updatemethod may be determined corresponding to the characteristics of thepartial image sequence (processing target) and the like. For example,when the correct answer position can be predicted to some extent, thebackward reference image may be shifted one by one since it is necessaryto mainly search the vicinity of the predicted position. When thecorrect answer position cannot be predicted, the above binary search orthe like may be used taking account of a reduction in the amount ofcalculations, for example.

When it has been determined that at least one of the (p+1)th to (q−1)thimages cannot be deleted, the processing section 100 may select the yth(y is an integer that satisfies y<q) image included in the backwardreference image selection interval as the next backward reference image.The processing section 100 may update the end point of the backwardreference image selection interval with the qth image.

This makes it possible to select the image that precedes the currentbackward reference image as the next backward reference image whenupdating the backward reference image. Since the search process is notlimited to a process that selects the adjacent image, the range thatprecedes the current backward reference image may not have beensearched, and may include a correct answer depending on the deletiondetermination result. In this case, it is possible to perform anappropriate process by performing a forward search process. The nextbackward reference image need not necessarily be selected from theadjacent image in the same manner as in the case of performing abackward search process.

When the jth (j is an integer) image corresponds to the end point of thebackward reference image selection interval, the processing section 100may set the value x based on the value (q+j)/2. Alternatively, when theith (i is an integer) image corresponds to the starting point of thebackward reference image selection interval, the processing section 100may set the value y based on the value (i+q)/2.

This makes it possible to use the binary search method when selectingthe next backward reference image. The image that is situated betweenthe current backward reference image and the end point is selected whenperforming a backward search process, and the image that is situatedbetween the current backward reference image and the starting point isselected when performing a forward search process. This makes itpossible to halve the search range (corresponding to the length of thebackward reference image selection interval). It is expected that theentire search range can be completely searched when log N images areselected as the backward reference image. Therefore, the amount ofcalculations can be reduced to N×log N. When N is very large, the amountof calculations can be significantly reduced as compared with the methodthat sequentially shifts the backward reference image in the backwarddirection (the amount of calculations is N²). Note that the value(q+j)/2 and the value (i+q)/2 are not necessarily an integer, and animage corresponding to each value may be absent. In such a case, themaximum integer that does not exceed the value (q+j)/2, or an integerthat is larger than the value (q+j)/2 by 1 may be used, for example.

The processing section 100 may perform a process that allows an imageamong a plurality of images selected as the forward reference image toremain in the summary image sequence when the starting point and the endpoint of the backward reference image selection interval are adjacent toeach other as a result of updating the starting point or the end pointof the backward reference image selection interval. The processingsection 100 may set one image among the plurality of images thatcorresponds to the starting point, and an image that follows the oneimage among the plurality of images that corresponds to the startingpoint in the partial image sequence to be a new partial image sequence,and process the new partial image sequence after setting the value p to1.

The expression “the starting point and the end point of the secondreference image selection interval are adjacent to each other” meansthat the image corresponding to the starting point and the imagecorresponding to the end point are adjacent to each other in the partialimage sequence. When N images have been set to be the partial imagesequence, it is considered that the partial image sequence is a set oftemporally or spatially continuous images. Therefore, the positionwithin the image sequence can be defined based on the continuity. Forexample, an image acquired at an earlier time precedes an image acquiredat a later time. Specifically, when the images included in the partialimage sequence are referred as first to Nth images, it is determinedthat an image is situated at a forward position when the number assignedto the image is small. Therefore, j=i+1 is satisfied when the ith imageand the jth (>i) mage included in the image sequence are adjacent toeach other.

This makes it possible to set a condition based on the starting pointand the end point of the backward reference image selection interval asa condition whereby the process on the partial image sequence isterminated. An image among the images that are determined to be OK whenselected as the backward reference image that is expected to be situatedfarthest from the forward reference image can be selected as the firstimage (corresponding to the next summary image) of the partial imagesequence by setting the termination condition. This is because thetermination condition is equivalent to the condition whereby theposition at which the image that is OK and the image that is NG areadjacent to each other is searched (see FIG. 28F, for example). Thismakes it possible to reduce the number of summary images included in thesummary image sequence that is output finally, and reduce the burdenimposed on the user, for example.

The first to seventh embodiments according to the invention and themodifications thereof have been described above. Note that the inventionis not limited to the first to seventh embodiments and the modificationsthereof. Various modifications and variations may be made of first toseventh embodiments according to the invention and the modificationsthereof without departing from the scope of the invention. A pluralityof elements described in connection with the first to seventhembodiments and the modifications thereof may be appropriately combinedto implement various configurations. For example, an arbitrary elementmay be omitted from the elements described above in connection with thefirst to seventh embodiments and the modifications thereof. Some of theelements described above in connection with different embodiments and/ormodifications may be appropriately combined. Any term cited with adifferent term having a broader meaning or the same meaning at leastonce in the specification and the drawings can be replaced by thedifferent term in any place in the specification and the drawings.Specifically, various modifications and applications are possiblewithout materially departing from the novel teachings and advantages ofthe invention.

What is claimed is:
 1. An image processing device comprising: an imagesequence acquisition section that acquires an image sequence thatincludes a plurality of images; and a processing section that performsan image summarization process that acquires a summary image sequencebased on a first deletion determination process and a second deletiondetermination process that delete some of the plurality of imagesincluded in the image sequence acquired by the image sequenceacquisition section, wherein: the processing section sets an attentionimage sequence that includes one attention image or a plurality ofattention images included in the plurality of images, selects a firstreference image from the attention image sequence, selects a firstdetermination target image from the plurality of images, and performsthe first deletion determination process that determines whether or notthe first determination target image can be deleted based on firstdeformation information that represents deformation between the firstreference image and the first determination target image, the processingsection sets a partial image sequence from the image sequence, aplurality of images that have been determined to be allowed to remain bythe first deletion determination process being consecutively arranged inthe partial image sequence, and the processing section selects a secondreference image and a second determination target image from the partialimage sequence, and performs the second deletion determination processthat determines whether or not the second determination target image canbe deleted based on second deformation information that representsdeformation between the second reference image and the seconddetermination target image.
 2. The image processing device as defined inclaim 1, wherein: the processing section performs at least one of afirst coverage ratio determination process and a first structuralelement determination process as the first deletion determinationprocess, the processing section performs at least one of a secondcoverage ratio determination process and a second structural elementdetermination process as the second deletion determination process, thefirst coverage ratio determination process is a process that calculatesa coverage ratio of the first determination target image by the firstreference image based on the first deformation information, anddetermines whether or not the first determination target image can bedeleted based on the calculated coverage ratio, the first structuralelement determination process is a process that determines whether ornot the first determination target image can be deleted based on resultsof a process that utilizes a structural element that corresponds to anattention area and the first deformation information, the secondcoverage ratio determination process is a process that calculates thecoverage ratio of the second determination target image by the secondreference image based on the second deformation information, anddetermines whether or not the second determination target image can bedeleted based on the calculated coverage ratio, and the secondstructural element determination process is a process that determineswhether or not the second determination target image can be deletedbased on results of a process that utilizes the structural element thatcorresponds to the attention area and the second deformationinformation.
 3. The image processing device as defined in claim 2,wherein: the processing section performs the second coverage ratiodetermination process as the second deletion determination process whenthe processing section performs the first coverage ratio determinationprocess as the first deletion determination process, and the processingsection performs the second structural element determination process asthe second deletion determination process when the processing sectionperforms the first structural element determination process as the firstdeletion determination process.
 4. The image processing device asdefined in claim 2, wherein: the processing section performs both thefirst coverage ratio determination process and the first structuralelement determination process as the first deletion determinationprocess.
 5. The image processing device as defined in claim 2, wherein:the processing section performs both the second coverage ratiodetermination process and the second structural element determinationprocess as the second deletion determination process.
 6. The imageprocessing device as defined in claim 2, wherein: the first coverageratio determination process is a determination process based on a resultof a comparison between a value that represents the coverage ratio ofthe first determination target image by the first reference image and afirst coverage ratio threshold value, the first structural elementdetermination process is a process that sets an element having a firstsize to be the structural element, and performs an erosion process thatutilizes the set structural element, or determines whether or not theset structural element is included in an area in which the firstdetermination target image is not covered by the first reference image,the second coverage ratio determination process is a determinationprocess based on a result of a comparison between a value thatrepresents the coverage ratio of the second determination target imageby the second reference image and a second coverage ratio thresholdvalue, and the second structural element determination process is aprocess that sets an element having a second size to be the structuralelement, and performs the erosion process that utilizes the setstructural element, or determines whether or not the set structuralelement is included in an area in which the second determination targetimage is not covered by the second reference image.
 7. The imageprocessing device as defined in claim 6, wherein: the processing sectionsets a value that differs from the first coverage ratio threshold valueto be the second coverage ratio threshold value.
 8. The image processingdevice as defined in claim 6, wherein: the processing section sets asize that differs from the first size to be the second size.
 9. Theimage processing device as defined in claim 1, wherein: when first toNth (N is an integer equal to or larger than 2) images are set to be thepartial image sequence, the processing section selects a forwardreference image and a backward reference image as the second referenceimage, the forward reference image being a pth (p is an integer thatsatisfies 1≦p≦N) image, and the backward reference image being a qth (qis an integer equal to or larger than p+2) image, and selects an rth (ris an integer that satisfies p+1≦r≦q−1) image as the seconddetermination target image, and the processing section calculatesforward deformation information that represents deformation between theforward reference image and the second determination target image, andbackward deformation information that represents deformation between thebackward reference image and the second determination target image, asthe second deformation information, and determines whether or not thesecond determination target image can be deleted based on the calculatedforward deformation information and the calculated backward deformationinformation.
 10. The image processing device as defined in claim 9,wherein: the processing section selects the backward reference imagefrom a backward reference image selection interval in which a startingpoint and an end point are set corresponding to (p+2)th to Nth images,and determines whether or not the second determination target image canbe deleted based on the forward reference image and the backwardreference image, and the processing section selects an xth (x is aninteger that satisfies x>q) image included in the backward referenceimage selection interval as a next backward reference image, and updatesthe starting point of the backward reference image selection intervalwith the qth image when it has been determined that (p+1)th to (q−1)thimages can be deleted.
 11. The image processing device as defined inclaim 10, wherein: the processing section selects a yth (y is an integerthat satisfies y<q) image included in the backward reference imageselection interval as the next backward reference image, and updates theend point of the backward reference image selection interval with theqth image when it has been determined that at least one image among the(p+1)th to (q−1)th images cannot be deleted.
 12. The image processingdevice as defined in claim 10, wherein: the processing section performsa process that allows an image among the plurality of images selected asthe forward reference image to remain in the summary image sequence whenthe starting point and the end point of the backward reference imageselection interval are adjacent to each other as a result of updatingthe starting point or the end point of the backward reference imageselection interval, and the processing section sets one image among theplurality of images that corresponds to the starting point, and an imageamong the plurality of images that follows the one image among theplurality of images that corresponds to the starting point in thepartial image sequence to be a new partial image sequence, and processesthe new partial image sequence after setting the value p to
 1. 13. Theimage processing device as defined in claim 1, wherein: the processingsection detects an attention area from the plurality of images, and setsan image among the plurality of images in which the attention area hasbeen detected to be the attention image.
 14. The image processing deviceas defined in claim 13, wherein: the image sequence acquisition sectionacquires a plurality of in vivo images as the image sequence, and theprocessing section detects a lesion area from the plurality of in vivoimages as the attention area, and sets an image among the plurality ofin vivo images in which the lesion area has been detected to be theattention image.
 15. The image processing device as defined in claim 1,wherein: the processing section performs the second deletiondetermination process on a plurality of the partial image sequences inparallel when the plurality of partial image sequences have been set.16. A computer-readable storage device having stored thereon a programthat is executable by a computer to cause the computer to function as:an image sequence acquisition section that acquires an image sequencethat includes a plurality of images; and a processing section thatperforms an image summarization process that acquires a summary imagesequence based on a first deletion determination process and a seconddeletion determination process that delete some of the plurality ofimages included in the image sequence acquired by the image sequenceacquisition section, wherein: the processing section sets an attentionimage sequence that includes one attention image or a plurality ofattention images included in the plurality of images, selects a firstreference image from the attention image sequence, selects a firstdetermination target image from the plurality of images, and performsthe first deletion determination process that determines whether or notthe first determination target image can be deleted based on firstdeformation information that represents deformation between the firstreference image and the first determination target image, the processingsection sets a partial image sequence from the image sequence, aplurality of images that have been determined to be allowed to remain bythe first deletion determination process being consecutively arranged inthe partial image sequence, and the processing section selects a secondreference image and a second determination target image from the partialimage sequence, and performs the second deletion determination processthat determines whether or not the second determination target image canbe deleted based on second deformation information that representsdeformation between the second reference image and the seconddetermination target image.
 17. An image processing method comprising:acquiring an image sequence that includes a plurality of images; settingan attention image sequence that includes one attention image or aplurality of attention images included in the plurality of images;selecting a first reference image from the attention image sequence, andselecting a first determination target image from the plurality ofimages; performing a first deletion determination process thatdetermines whether or not the first determination target image can bedeleted based on first deformation information that representsdeformation between the first reference image and the firstdetermination target image; setting a partial image sequence from theimage sequence, a plurality of images that have been determined to beallowed to remain by the first deletion determination process beingconsecutively arranged in the partial image sequence; selecting a secondreference image and a second determination target image from the partialimage sequence; performing a second deletion determination process thatdetermines whether or not the second determination target image can bedeleted based on second deformation information that representsdeformation between the second reference image and the seconddetermination target image; and performing an image summarizationprocess that deletes some of the plurality of images included in theimage sequence based on the first deletion determination process and thesecond deletion determination process to acquire a summary imagesequence.